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AN  APPROACH  FOR  EFFICIENTLY 
MANAGING  DOD  RESEARCH  AND 
DEVELOPMENT  PORTFOLIOS 

Drs.  Carolyn  Wong,  Paul  Steinberg, 

Kenneth  Horn,  and  Elliot  Axelband 


Managing  Department  of  Defense  (DoD)  research  and  development  (R&D) 
portfolios  is  a  challenging  task  today.  Not  only  do  defense  R&D  managers 
have  limited  resources  to  spend  on  pursuing  new  technologies,  but  there  is 
also  an  active  civil  and  commercial  market  for  some  technologies  the  DoD  is 
interested  in.  How  can  the  DoD  better  understand  what  areas  it  must  pursue 
on  its  own  and  what  areas  it  might  be  better  off  collaborating  with  non-DoD 
organizations?  This  article  presents  a  straightforward  approach  for  managing 
DoD  R&D  portfolios  that  can  help  DoD  managers  understand  how  their  R&D 
efforts  are  allocated,  and  how  they  might  more  efficiently  manage  them  to 
take  advantage  of  scarce  resources  and  technological  capabilities  elsewhere. 
After  describing  the  approach,  the  authors  illustrate  it  using  the  example  of 
the  basic  research  technologies  part  of  the  U.S.  Army’s  R&D  portfolio. 


To  maintain  its  technological  edge, 
the  Department  of  Defense  (DoD), 
through  its  Armed  Services  and 
agencies,  spent  nearly  $9.3  billion  in  fiscal 
year  1995  on  basic,  exploratory  develop¬ 
ment,  and  advanced  development  research. 
While  in  the  past,  the  DoD  accepted  such 
research  and  development  (R&D)  invest¬ 
ments  simply  as  necessary  expenditures, 
today,  the  Do D  faces  a  series  of  demands 
and  constraints  that  argue  for  more  carefully 
and  more  efficiently  managing  that  invest¬ 
ment:  These  include: 


future  reductions  in  science  and  tech¬ 
nology  (S&T)  funding — reductions 
that  have  averaged  15  percent  per  year 
over  the  past  few  years; 

commercial  domination  of  many  of  the 
important  technological  areas  for  the 
DoD,  such  as  information  technologies; 

growth  in  international  technology  ca¬ 
pabilities  and  in  competition  from  Eu¬ 
ropean  and  Japanese  companies;  and 
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•  a  changing  research  climate  within  the 
government,  with  a  growing  ideologi¬ 
cal  shift  away  from  big  government 
involvement  in  R&D. 

One  of  the  constraints  affecting  DoD 
research  investments  is  clearly  resources: 
Resources  to  conduct  DoD  R&D  are  sim¬ 
ply  more  limited  than  they  were  in  the  past. 
But  even  if  the  DoD  had  unlimited  re¬ 
sources  with  which  to  pursue  its  R&D 
projects,  it  is  not  always  clear  that  the  DoD 
is  in  the  best  position  to  lead  in  certain 
technological  areas.  There  is  a  very  ac¬ 
tive  civil  and  commercial  market,  both 
domestic  and  international,  that  is  pursu¬ 
ing  its  own  R&D  activities  in  the  same 
areas  as  is  the  DoD,  and  in  many  situa¬ 
tions,  the  commercial  and  civil  sectors  are 
technologically  ahead. 

Take,  for  example,  global  positioning 
system  (GPS)  technology.  GPS  began  as 
an  Air  Force  program  to  put  up  satellites 
to  generate  radio  navigation  signals  to 
enhance  the  navigational  capability  of 
military  vehicles  and  guided  weapons.  The 
DoD  R&D  effort  in  GPS  technology 
spanned  decades  and  consumed  billions 
of  defense  dollars.  Technical  advances  by 


the  DoD  ultimately  led  to  recognition  of 
commercial  applications  and  to  DoD’s 
decision  to  provide  a  commercially  avail¬ 
able  channel  in  addition  to  the  secure  mili¬ 
tary  one.  During  the  development  of  GPS 
technology,  advances  in  semiconductor 
technology  enabled  small  receivers  to  be 
built,  and  the  market  for  geographic  in¬ 
formation  systems  grew  rapidly.  These 
changes  led  to  a  significant  commercial 
R&D  investment  in  GPS  technology. 

The  end  result  is  that  the  commercial 
sector  now  spends  more  than  the  DoD  in 
a  technological  area  that  the  DoD  pio¬ 
neered  and  once  led.  In  terms  of  R&D 
projects,  the  DoD  now  has  the  opportu¬ 
nity  to  collaborate  with  commercial  firms 
in  areas  of  common  GPS  interest  (e.g.,  the 
ground-based  segment  of  GPS),  thus  re¬ 
ducing  DoD’s  need  to  develop  technology 
available  from  commercial  sources. 

Of  course,  not  all  technological  areas 
are  like  GPS  technology.  For  example, 
R&D  in  high-energy  lasers  and  radiation- 
tolerant  semiconductors  for  ballistic  mis¬ 
sile  defense  spacecraft  are  DoD-driven 
and  unlikely  to  change,  since  purely 
commercial  firms  have  little  interest  in 
pursuing  research  in  these  areas. 


Carolyn  Wong  is  currently  an  operations  research  specialist  at  RAND.  She  holds  a  Ph.D.  degree 
in  electrical  engineering  (operations  research),  an  M.S.  degree  in  management,  and  a  B.A. 
degree  in  mathematics,  all  from  the  University  of  California  at  Los  Angeles. 

Paul  Steinberg  is  currently  a  communications  analyst  at  RAND.  He  holds  a  Ph.D.  degree  in 
English  from  the  University  of  California  at  Berkeley. 

Kenneth  Horn  is  currently  director  of  the  Force  Development  and  Technology  Program  in  the 
RAND  Arroyo  Center.  He  holds  a  Ph.D.  degree  in  aeronautics  and  astronautics  from  Stanford 
University,  and  M.S.  and  B.A.  degrees  in  mechanical  engineering  from  Rice  University. 
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California  at  Los  Angeles. 
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How  can  the  DoD  manage  its  massive 
portfolio  of  R&D  projects  in  a  way  that 
enables  it  to  understand  what  areas  it  must 
pursue  on  its  own?  And  in  what  areas 
might  it  be  better  off  collaborating  with 
commercial  firms  and  leveraging  off  their 
capabilities?  Here  we  present  an  approach 
for  managing  DoD  R&D  portfolios  that 
can  help  DoD  managers  understand  how 
their  R&D  efforts  are  allocated  and  how 
they  might  more  efficiently  manage  them 
to  take  advantage  of  scarce  resources  and 
technological  capabilities  elsewhere.  Af¬ 
ter  describing  the  approach,  we  illustrate 
it  using  the  example  of  the  U.S.  Army’s 
R&D  portfolio,  looking  particularly  at  its 
basic  research  technologies. 


The  DoD  R&D  Assessment  Matrix: 

A  Tool  for  Managing  R&D  Projects 


In  thinking  about  a  useful  way  to  help 
DoD  research  managers  categorize  vary¬ 
ing  projects  within  an  R&D  portfolio  and 
manage  them  effectively,  we  developed  an 
approach  that  involves  using  a  two-dimen¬ 
sional  matrix  that  is  partitioned  into  four 
management  domains  (Saunders  et  al., 
1995;  Wong,  1998).1  This  matrix  is  de¬ 
signed  to  serve  as  a  tool  to  obtain  a  first- 
order  indication  of  which  defense  tech¬ 
nologies  might  overlap  with  commercial 
technologies.2  Below,  we  discuss  the  two 
dimensions  and  the  four  management 
domains  in  more  detail. 

Matrix  Dimensions 

Two  dimensions  are  critical  to  cat¬ 
egorizing  any  DoD  R&D  technology 
area:  the  technology’s  utility  to  a  mili¬ 
tary  Service  or  DoD  agency,  and,  since 
many  DoD  technologies  are  being  pursued 


"The  market 
breadth  dimension 
is  designed  to  indi¬ 
cate  the  level  of 
interest  outside 
DoD  in  a  particular 
technology;" 


by  commercial  firms,  the  technology’s 
market  breadth. 

The  Service  or  DoD  agency  utility 
dimension  reflects  the  potential  contribu¬ 
tion  of  the  technology  to  helping  the  Ser¬ 
vice  or  DoD  agency  accomplish  its  mis¬ 
sion.  In  our  framework.  Service  or  DoD 
agency  utility  is  represented  as  a  continu¬ 
ous  scale  that  ranges  from  low  to  high. 
An  example  of  a  technology  that  would 
have  low  Service  or  DoD  agency  utility  is 
one  that  is  not  expected  to  contribute  di¬ 
rectly  to  maintaining  DoD’s  future  defense 
capability.  On 
the  other  hand,  a 
technology  that 
is  critical  to  con¬ 
ducting  future 
defense  opera¬ 
tions  is  an  ex¬ 
ample  of  a  high 
Service  or  DoD 
agency  utility  -r 
technology.3 

The  market  breadth  dimension  is 
designed  to  indicate  the  level  of  interest 
outside  DoD  in  a  particular  technology.  If 
a  technology  has  many  potential  govern¬ 
ment  and  commercial  uses  (i.e.,  everybody 
wants  it),  then  industry’s  interest  is  likely 
to  be  higher  than  if  the  technology  had  po¬ 
tential  use  only  for  a  particular  military 
Service  or  DoD  agency  (Service-  or  DoD 
agency-unique).  Industry’s  interest  in  the 
former  case  is  likely  to  be  higher,  since 
advances  in  the  technology  have  poten¬ 
tial  uses  in  many  products  or  services. 
Hence,  industry  is  likely  to  perceive  such 
a  technology  as  one  that  is  more  likely  to 
result  in  higher  profits.  In  our  framework, 
market  breadth  is  represented  as  a  con¬ 
tinuous  scale  that  ranges  from  a  technol¬ 
ogy  having  potential  uses  to  a  particular 
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military  Service  or  DoD  agency  only  (Ser¬ 
vice-  or  DoD  agency-unique),  to  potential 
government  uses  only,  to  potential  govern¬ 
ment  and  commercial  uses  (generic). 

Defining  Management  Domains 

Technologies  that  have  a  moderate  to 
high  utility  rating  will  fall  into  the  upper 
half  of  the  matrix  framework.  These  tech¬ 
nologies  are  generally  vital  to  the  success¬ 
ful  completion  of  a  Service  or  DoD 
agency’s  mission.  The  Service  or  DoD 
agency  will  most  likely  want  to  be  active 


and  maintain  some  control  (e.g.,  through 
funding)  over  the  R&D  activities  that  oc¬ 
cur  in  these  technologies.  At  the  same 
time,  technologies  whose  market  breadth 
is  limited  to  the  government  will  fall  into 
the  left-hand  side  of  the  matrix.  The  gov¬ 
ernment  is  less  likely  to  find  suitable  in¬ 
dustrial  partners  in  these  technologies  be¬ 
cause  commercial  interest  is  limited. 
Hence,  government  funding  is  likely  to  be 
required  for  R&D  to  occur  in  technolo¬ 
gies  that  fall  roughly  in  the  region  called 
the  lead  domain  in  Figure  1 . 


Figure  1.  Management  Domains  Within  Assessment  Matrix 
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Since  industry  interest  is  required  for 
collaboration  to  be  successful,  the  right- 
hand  region  of  the  matrix  framework, 
where  industry  interest  is  moderate  to 
high,  is  the  general  area  where  collabora¬ 
tion  is  likely  to  work.  Under  tight  fiscal 
constraints,  however,  a  Service  or  DoD 
agency  may  find  that  it  must  dramatically 
lower  or  eliminate  its  R&D  funding  in 
some  technologies.  We  argue  that  the  can¬ 
didates  for  reduced  funding  are  those  with 
low  Service  or  DoD  agency  utility — that 
is,  those  in  the  lower  half  of  the  frame¬ 
work.  Among  those  with  low  utility,  those 
with  high  industry  interest  (i.e.,  technolo¬ 
gies  in  the  right-hand  side  of  the  frame¬ 
work)  may  be  better  candidates  for  low¬ 
ered  funding  because  R&D  is  more  likely 
to  be  continued  in  those  technologies 
through  industry  funding.  R&D  activities 
are  unlikely  to  continue  in  Service-  or 
DoD  agency-unique  technologies  without 
government  funding.  Hence,  technologies 
that  are  the  best  candidates  for  reduced 
funding  fall  roughly  in  the  region  called 
the  monitor  domain  in  Figure  1 . 

The  remaining  area  on  the  framework 
in  Figure  1  is  divided  into  two  domains: 
initiate  and  participate.  Technologies  that 
fall  into  the  initiate  domain  have  higher 
utility;  hence,  the  government  may  want 
to  spend  more  funds  and  effort  to  ensure 
that  research  in  these  technologies  accom¬ 
plishes  government  goals.  Technologies 
that  fall  into  the  participate  domain  have 
lower  utility;  as  a  result,  the  government 
may  be  more  willing  to  compromise  on  a 
set  of  goals  in  exchange  for  the  benefits  a 
collaborative  effort  would  bring. 

The  Services  and  DoD  agencies  have 
traditionally  judged  the  progress  and  suc¬ 
cess  of  R&D  efforts  in  terms  of  three 
benchmarks— performance,  schedule,  and 


cost  and  budget4  characteristics.  As  Fig¬ 
ure  1  shows,  our  framework  comprises 
four  management  domains,  which  reflect 
different  approaches  to  managing  the  three 
benchmarks.5  Our  discussion  below  is 
presented  from  DoD’s  viewpoint. 


Lead  Domain 

For  those  technologies  in  the  lead  do¬ 
main,  the  Service  or  DoD  agency  should 
expect  to  spend  its  own  funds  to  realize 
the  benefits  of 


"For  those  tech¬ 
nologies  in  the 
lead  domain,  the 
;  Service  or '  Dob '  •  •'  - 
agency  should 
expect  to  spend  its 
own  funds  to  realize 
the  benefits  of 
technological 
advances*" 


technological 
advances . 6 
When  a  Service 
or  DoD  agency 
chooses  to  lead, 
it  defines  the 
performance 
goals,  provides 
the  vision,  and 
specifies  the  po¬ 
tential  products  J  > 
or  capabilities.  The  Service  or  DoD 
agency  also  sets  forth  its  schedule  require¬ 
ments  and  sets  the  resource  (cost  and  bud¬ 
get)  constraints.  Under  a  Service-  or  DoD 
agency-lead  management  approach,  the 
DoD  group  uses  contractual  measures  to 
control  performance,  schedule,  and  use  of 
resources  (budget).  In  the  lead  role,  the 
DoD  group  is  conducting  “business  as 
usual,”  and  the  contracted  activities  are 
usually  not  collaborative. 


Initiate  Domain 

Technologies  that  fall  into  the  initiate 
domain  have  high  DoD  agency  or  mili¬ 
tary  Service  utility  and  high  industry  in¬ 
terest.  Although  industry  might  be  fund¬ 
ing  R&D  in  these  technologies,  the  DoD 
might  not  wish  to  just  stand  by  and  rely 
totally  on  industry  to  meet  DoD  goals. 
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Instead,  a  Service  or  DoD  agency  may 
want  to  actively  seek  and  initiate  collabo¬ 
rative  R&D  efforts  to  ensure  that  R&D  in 
these  areas  addresses  its  goals. 

When  a  Service  or  DoD  agency  chooses 
to  initiate,  it  defines  its  performance  goals, 
provides  its  vision,  and  specifies  products 
and  capabilities  of  DoD  interest.  The  Ser¬ 
vice  or  DoD  agency  may  also  set  forth  its 
schedule  requirements  and  set  its  resource 
(cost  and  budget)  constraints. 

In  its  search  for  a  collaborating  partner 
in  industry,  the  Service  or  DoD  agency 
may  look  for  areas  of  intersection  among 
its  performance  goals  and  those  of  poten¬ 
tial  industrial  partners,  rather  than  negoti¬ 
ate  a  set  of  goals,  since  these  technolo¬ 
gies  are  primarily  of  high  Service  or  DoD 
agency  utility.  The  DoD  group  may  also 
look  for  compatible  schedules  or  negoti¬ 
ate  an  accept- 

"Under  a  DoD-  able  schedule 

initiate  manage-  and  may  also 

ment  approach,  negotiate  a  set 

the  Service  or  DoD  of  resource  con¬ 


agency  tries  to 


straints.  Under 


control  performance, 
but  may  share  con¬ 
trol  of  schedule  and 
use  of  resources/' 


a  DoD-initiate 
management 
approach,  the 
Service  or  DoD 


agency  tries  to 
control  performance,  but  may  share  con¬ 
trol  of  schedule  and  use  of  resources.  In 
the  initiate  role,  the  DoD  group  is  collabo¬ 
rating  with  industry.  In  such  an  effort,  it 
should  achieve  the  same  or  nearly  all  the 
same  goals  it  would  achieve  if  it  conducted 
the  activity  without  collaboration. 


Participate  Domain 

Technologies  in  the  participate  domain 
have  moderate  market  breadth  and 
moderate  Service  or  DoD  agency  utility. 


Under  tight  fiscal  constraints,  neither  the 
Service,  DoD  agency,  nor  industry  will 
have  enough  funds  to  invest  much  in  these 
technologies.  Collaboration  may  allow  the 
Service  or  DoD  agency  and  industry  to 
pool  resources  to  perform  R&D  in  these 
areas.  However,  the  Service  or  DoD 
agency  may  not  want  to  expend  additional 
efforts  to  actively  seek  and  initiate  re¬ 
search  activities.  For  technologies  in  this 
domain,  both  the  Service  or  DoD  agency 
and  industry  can  design  and  participate  in 
activities  for  mutual  benefit.  Such  efforts 
may  require  both  the  DoD  group  and  in¬ 
dustry  to  compromise  on  a  set  of  R&D 
goals.  Without  a  willingness  to  adjust 
goals,  a  joint  investment  may  not  be  at¬ 
tractive  enough  to  the  Service  or  DoD 
agency  or  potential  industry  partners. 

When  a  Service  or  DoD  agency  chooses 
to  participate,  it  may  negotiate  acceptable 
performance  goals  if  it  cannot  find  an  ap¬ 
propriate  intersection  with  industry  perfor¬ 
mance  goals.  The  Service  or  DoD  agency 
may  also  negotiate  an  acceptable  schedule 
as  well  as  a  set  of  resource  constraints.  Un¬ 
der  a  DoD-participate  management  ap¬ 
proach,  the  Service  or  DoD  agency  has 
shared  control  of  performance,  schedule, 
and  use  of  resources.  In  the  participate 
role,  the  Service  or  DoD  agency  is  col¬ 
laborating  with  industry  and  should 
achieve  at  least  some  of  the  same  goals  it 
would  achieve  if  it  conducted  the  activity 
without  collaboration. 

Monitor  Domain 

Technologies  in  the  monitor  domain 
have  low  Service  or  DoD  agency  utility 
(i.e.,  do  not  contribute  directly  or  very 
much  to  its  overall  mission)  and  high  mar¬ 
ket  breadth.  Under  tight  fiscal  constraints, 
the  Service  or  DoD  agency  may  have  to 
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Table  1.  Benchmark  Characteristics  ef  Management  Domains 


Management 

Domain 

Characteristics 

•  Performance 

•  Schedule 

•  Resources 

Effects 

•  Control 

•  Collaboration 

Lead 

•  Define  performance  goals, 
vision,  products,  capabilities 

•  Set  schedule 

•  Set  resource  constraints 
(e.g.,  budget) 

•  Service/DoD  agency  has  full 
control  and  responsibility  for 
performance,  schedule,  and 
use  of  resources  (e.g.,  budget) 

•  Usually  not  collaborative 

Initiative 

•  Define  performance  goals, 
vision,  products,  capabilities 

•  Set  or  negotiate  acceptable 
schedule 

•  Set  or  negotiate  resource 
constraints  (e.g.,  budget) 

•  Service/DoD  agency  controls 
performance,  but  shares 
control  of  schedule  and 
sometimes  resources 

(e.g.,  budget) 

•  Can  be  collaborative 

Participate 

•  Define  key  performance 
requirements  and  negotiate 
performance  goals 

•  Negotiate  acceptable  schedule 

•  Negotiate  resource  constraints 
(e.g.,  Army’s  share  of  budget) 

•  Service/DoD  agency  shares 
control  of  performance, 
schedule,  and  resources 

•  Should  be  collaborative 

Monitor 

•  Vigilant  communication  of 
performance  requirements 

•  Communication  of  schedule 
requirements 

•  Little  or  no  resource 
commitments 

•  Service/DoD  agency  has  no 
control  of  performance, 
schedule  of  resources 

•  Service/DoD  agency  does  not 
have  a  formal  role 

let  industry  take  the  lead  for  technologies 
in  the  monitor  domain  and  limit  its  own 
R&D  investment  there,  restricting  its  role 
to  one  of  active  monitoring.  Active  moni¬ 
toring  could  include  low  or  no-cost  activi¬ 
ties,  such  as  establishing  working  relation¬ 
ships  with  industry  leaders,  regularly  (but 
informally)  communicating  DoD  needs  in 
the  technology,  and  attending  workshops 
and  conferences.  In  the  monitor  role,  the 
Service  or  DoD  agency  does  not  have  a 
formal  role  and  has  no  control  over  per¬ 
formance,  schedule,  or  use  of  resources 
committed  to  R&D  activities. 


Table  1  summarizes  typical  perfor¬ 
mance,  schedule,  and  cost  benchmark 
characteristics  and  control  and  collabora¬ 
tion  effects  of  the  four  management  ap¬ 
proaches  in  the  matrix.  As  noted  earlier,  the 
demarcations  between  the  management 
approaches  are  fuzzy. 

Applying  the  Approach:  The  Example 
of  the  Army's  Basic  Research  Projects 

To  show  how  the  approach  works,  we 
use  the  Army  as  the  Service  or  DoD 
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agency,  focusing  on  the  Army’s  basic 
(6.1)  research  projects  as  an  example 
application.7  As  such,  we  use  the  list  of 
technologies  found  in  the  Army’s  fiscal 
year  1995  Research,  Development,  Test, 
and  Evaluation  (RDT&E)  Program 
(known  as  the  R-l)  under  the  basic  re¬ 
search  heading  (OCDoD,  1994).  We  lim¬ 
ited  our  considerations  to  those  technolo¬ 
gies  that  showed  a  positive  funding  level 
for  fiscal  year  1995  (proposed).  The  list 
of  Army  basic  research  technologies  is 
shown  in  the  box  on  page  347.8 


How  the  Basic  (6.1)  Research 
Techhologies  Were  Placed 
Using  the  Approach 


Using  the  technologies  shown  in  the 
box,  we  placed  them  on  the  matrix  assess¬ 
ment  framework  with  the  help  of  a  group 
of  researchers  with  backgrounds  in  engi¬ 
neering,  operations  research,  business 
management,  and  the  physical  sciences. 
Experience  levels  ranged  from  five  years 
to  decades  of  experience  in  R&D  issues. 

Every  researcher 
"For  computes-  had  worked  on 

fional  convenience.  Army  R&D 
researchers  assumed  projects  for  at 


a  scale  off  zero  to 
three  ffor  the  market 
breadth  axis  and  ffor 
the  Army  utility 
axis." 


least  several 
years,  and  all 
were  familiar 
with  the  Army’s 
current  R&D 
program.  In  ad¬ 
dition,  most  researchers  had  experience 
with  commercial  firms  that  did  business 
with  the  Army. 


Each  researcher  was  furnished  with 
descriptive  material  on  the  technologies. 
To  minimize  the  influence  of  current 


budget  allocations  on  the  placement  of 
technologies  on  the  framework,  no  bud¬ 
get  information  was  included  in  the  de¬ 
scriptive  material,  nor  was  it  discussed  or 
analyzed  until  the  conclusion  of  iterative 
discussions  to  resolve  differences  in  opin¬ 
ion  on  where  some  technologies  should  be 
placed. 

Each  researcher  also  received  guidance 
on  how  to  interpret  the  endpoints  of  the 
market  breadth  and  Army  utility  dimensions 
of  the  framework.9  For  the  market  breadth 
axis,  placement  on  the  left-most  portion 
of  the  framework  indicated  “close  to  Army 
unique”  and  placement  in  the  right-most 
portion  indicated  “close  to  government 
and  commercial  uses”  (generic).  For  Army 
utility,  placement  on  the  lower  portion  of 
the  framework  indicated  that  the 
technology’s  potential  contribution  to  ac¬ 
complishment  of  the  Army’s  mission  is 
low  or  small.  For  example,  technologies 
that  do  not  directly  contribute  to  maintain¬ 
ing  future  combat  capabilities  should  be 
placed  near  the  bottom.  Placement  at  the 
top  of  the  framework  indicates  that  the 
potential  contribution  the  technology  is 
expected  to  make  is  great  (e.g.,  technolo¬ 
gies  that  are  critical  to  future  combat 
effectiveness). 

For  computational  convenience,  re¬ 
searchers  assumed  a  scale  of  zero  to  three 
for  the  market  breadth  axis  and  for  the 
Army  utility  axis.  A  market  breadth  value 
of  zero  indicated  potential  Army  uses  only 
(Army  unique),  and  a  market  breadth 
value  of  three  represented  potential  gov¬ 
ernment  and  commercial  uses  (generic). 
Similarly,  an  Army  utility  value  of  zero 
indicated  low  Army  utility,  and  a  value  of 
three  indicated  high  Army  utility. 

For  each  basic  research  technology 
shown  in  the  adjacent  box,  each  researcher 
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Army  Basic  (6.1)  Research  Technologies  in  the  R-l 


Artificial  intelligence  technology 


Aviation  technology 


Ballistics  technology 

Chemical,  smoke,  and  equipment  defeating  technology 
Combat  vehicle  and  automotive  technology 


Command,  control,  and  communications  technology 


Computer  and  software  technology 
Electronic  survivability  and  fuzing  technology 


Electronic  warfare  technology 


Electronics  and  electronic  devices 


Environmental  quality  technology 
Human  factors  engineering  technology 


Joint  services  small  arms  program 


Laser  weapons  technology 


Logistics  technology 
Manpower/personnel/training  technology 


Materials  technology 


Medical  technology 
Military  engineering  technology 
Missile  technology 


Modeling  and  simulation 


Night  vision  technology 
Nonsystem  training  device  technology 
Weapons  and  munitions  technology 


Note:  Army  Basic  Research  includes  four  classified  programs  not  shown  in  the  table. 
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specified  a  market  breadth  value  and  an 
Army  utility  value.  Each  researcher 
worked  independently  to  establish  his  ini¬ 
tial  values,  with  one  researcher  tabulating 
the  results.  The  tabulated  results  showed 
consensus  in  most  technology  areas.  For 
example,  all  researchers  specified  values 
for  environmental  quality  technology  that 
placed  this  technology  in  the  monitor 
domain.  Similarly,  all  researchers  speci¬ 
fied  values  for  medical  technology  and 
computers  that  placed  these  technologies 

in  the  initiate 
"Ta  resolve  these '  tf  i-  domain.  In  ad- 
differences,  the  dition,  all  re¬ 
researchers  held  a  searchers  speci- 
series  ef  discussions  fied  values  for 
to  try  to  reach  a  all  classified 
consensus."  programs  that 

placed  these 
technologies  in  the  lead  domain.  However, 
there  were  also  some  technologies  for 
which  there  was  no  initial  consensus.  For 
example,  some  researchers  viewed  night 
vision  as  a  lead  domain  technology,  while 
others  felt  that  it  was  an  initiate  technology. 

To  resolve  these  differences,  the 
researchers  held  a  series  of  discussions  to 
try  to  reach  a  consensus.  For  this  exercise, 
we  considered  “widely  different  values” 
to  be  values  that  differed  by  more  than  0.5 
and  placed  the  technology  in  a  different 
domain.  The  discussions  methodically 
moved  from  one  technology  to  the  next, 
but  the  placement  for  some  technologies 
required  multiple  discussions.  The  discus¬ 
sions  took  place  once  a  week  for  about  a 
month.  For  this  exercise,  we  viewed  val¬ 
ues  that  would  place  the  technology  in  the 
same  management  domain  as  a  consen¬ 
sus.  Hence,  researchers  could  adjust  their 
specification  of  values  to  reach  consen¬ 
sus  but  still  have  some  leeway  to  express 


their  opinion  about  where  the  technology 
should  be  placed  on  the  matrix. 

The  discussions  resulted  in  modified 
values  for  some  of  the  technologies  by 
some  researchers.  Any  remaining  discrep¬ 
ancies  were  adjudicated  by  the  group 
leader.  After  the  discussions,  we  averaged 
the  market  breadth  values  and  the  Army 
utility  values  for  each  technology  in  each 
category.  The  pair  of  average  values  for 
each  technology  in  each  category  deter¬ 
mined  the  technology’s  placement  on  the 
framework. 

Results  of  Placing  the 

Basic  (6.1)  Research  Technologies 

Usiho  the  Approach _ 

Figure  2  shows  the  end  result  of  the 
exercise  of  placing  the  Army’s  basic  re¬ 
search  technologies.  We  clearly  see  that 
the  Army’s  basic  research  R&D  is  not  a 
mass  of  homogeneous  technologies.  Of 
the  24  technologies  considered  (not  count¬ 
ing  the  classified  programs),  only  6 — one 
quarter — are  categorized  as  having  both 
a  high  Army  utility  and  as  Army  unique 
in  terms  of  market  interest.  Half  of  the 
technologies  are  both  of  high  Army  util¬ 
ity  and  of  interest  to  nondefense  industries, 
and  another  fifth  of  the  technologies  are 
both  of  moderate  Army  utility  and  mod¬ 
erate  interest  to  industry.  Only  one  tech¬ 
nology  is  of  low  utility  to  the  Army  and 
high  interest  to  industry. 

This  distribution  of  technologies  shows 
that  it  may  make  sense  for  managers  to 
take  different  approaches  in  managing  the 
technologies.  For  example,  those  six  tech¬ 
nologies  in  the  lead  category  are  basically 
core  Army  R&D,  technologies  that  are  not 
of  interest  to  industries  other  than  some 
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High 


Army 

Utility 


Joint  Service  small  arms  program 
Materials  technology 
Military  engineering  technology 
Non-system  training  device  technology 


Low 

Army  Unique  Generic 

Market  Breadth 

Figure  2.  Placement  of  Basic  (6.1)  Research  Technologies 


military  suppliers  and  ones  that  the  Army  collaboration  with  industry,  in  some  cases 
likely  has  a  strong  technological  lead  in.  to  save  resources  and  in  some  cases  to  le- 
These  technologies  will  require  complete  verage  off  technological  leads  held  by 
Army  funding  and  can  be  managed  using  industry. 

basic  contract  mechanisms  with  traditional  In  fact,  the  Army  has  already  started 
military  suppliers.  collaborative  efforts  to  develop  some  tech- 

However,  the  17  technologies  in  the  nologies  that  fall  into  the  initiate  and  par- 
initiate  and  participate  categories — which  ticipate  domains.  For  example,  in  Project 

together  represent  nearly  75  percent  of  the  Plowshares,  the  Orange  County,  FL,  lo- 
total — are  technologies  in  which  the  Army  cal  government  is  using  Army-generated 

and  nontraditional  military  suppliers  in  the  computer  simulations  to  aid  in  disaster 
commercial  world  have  a  mutual  interest.  relief.  And  the  Army  Tank- Automotive 
Such  technologies  offer  the  potential  for  and  Armaments  Command  (TACOM)  has 


Initiate 


Ballistics  technology 
Electronic  survivability  and 
fuzing  technology 
Electronic  warfare  technology 
Missile  technology 
Night  vision  technology 
Weapons  and  munitions 
technology 

Four  classified  programs 


Artificial  intelligence  technology 
Aviation  technology 

Chemical,  smoke  and  equipment  defeating 
technology 

Combat  vehicle  and  automotive  technology 
Command,  control,  communications 
technology 

Computers  and  software  technology 
Electronics  and  electronic  devices 
Laser  weapons  technology 
Logistics 

Manpower/personnel/training  technology 
Medical  technology 
Modeling  and  simulation 


Participate 

Human  factors  engineering  technology 


Monitor 
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collaborated  with  the  Big  Three  automo¬ 
bile  manufacturers  to  form  the  National 
Automotive  Center  (NAC).  Army  and 
industry  collaborative  efforts  are  also 
ongoing  in  the  information  technology  area. 

Managing  such  collaborations  will  re¬ 
quire  something  other  than  the  standard 
contracts  used  for  traditional  military  sup¬ 
pliers.  To  this  end,  cooperative  agreements 
(CAs)  and  especially  other  transactions 
(OTs)  are  instruments  more  suitable  for 
collaborative  efforts.  In  1989,  Congress 
authorized  CAs  (in  10  U.S.C.  §2358)  and 
OTs  (in  10  U.S.C.  §2371)  for  use  by  the 
Defense  Advanced  Research  Projects 


Agency  (DARPA)  as  alternative  mecha¬ 
nisms  for  conducting  R&D.  Authority  tc 
use  CAs  and  OTs  was  extended  to  all  ol 
DoD,  including  the  military  Services,  in 
1991. 9  U.S.  Code  §2371  includes  the  cat¬ 
egory  of  “other  transactions”  as  an  essen¬ 
tially  undefined 

"•••fhere  was  con¬ 


sensus  that  environ¬ 
mental  technology 
belonged  in  the 
monitoring  category; 
since  developing  the 
technology  is  clearly 
not  central  to  the 
Army's  warfighting 
mission  and  the 
technology  itself 
is  well  developed 
by  commercial 
companies/' 


term.  DARPA 
has  interpreted 
Section  2371  to 
mean  that  OTs 
are  a  class  of 
transactions 
outside  the  pro¬ 
curement  and 
assistance  cat¬ 
egories,  and 
DARPA  has 
implemented 
them  as  such 


since  1989,  the 
time  of  the  statute’s  original  enactment. 

As  mentioned  above,  there  was  consen¬ 
sus  that  environmental  technology  be¬ 
longed  in  the  monitoring  category,  since 
developing  the  technology  is  clearly  not 
central  to  the  Army’s  warfighting  mission 
and  the  technology  itself  is  well  developed 


by  commercial  companies.  While  the 
Army  will  need  to  use  the  technology  to 
deal  with  the  environmental  problems  it 
faces  on  military  bases,  staying  in  touch 
with  what  is  going  on  in  industry  and 
outsourcing  as  appropriate  would  seem  to 
make  more  sense  than  developing  the 
technology  in-house. 

Although  not  shown  here,  the  distribu¬ 
tion  of  the  Army’s  exploratory  develop¬ 
ment  (6.2)  technologies  shifts  toward  the 
left  side  of  the  framework  in  the  direction 
of  the  lead  domain:  lead,  30  percent;  ini¬ 
tiate  and  participate,  70  percent.  This  find¬ 
ing  indicates  that  in  addition  to  the  col¬ 
laborative  opportunities  in  basic  research 
technologies,  the  Army  also  has  many 
potential  opportunities  to  collaborate  with 
industry  in  exploratory  development 
technologies. 

The  distribution  for  advanced  develop¬ 
ment  (6.3)  technologies  shows  a  more 
marked  shifted  toward  the  left  side  of  the 
framework:  lead,  70  percent;  initiate  and 
participate,  26  percent;  monitor,  4  percent. 
This  finding  is  consistent  with  expecta¬ 
tions.  As  a  technology  progresses  from 
basic  research,  to  exploratory  develop¬ 
ment,  to  advanced  development,  and  on 
to  an  identifiable  product,  the  research  be¬ 
comes  more  specific  in  terms  of  its  mili¬ 
tary  application;  hence,  fewer  collabora¬ 
tive  research  opportunities  with  industry 
would  be  expected.  Therefore,  the  use  of 
the  matrix  evaluation  tool  correctly  indi¬ 
cated  the  expected  decline  of  collabora¬ 
tive  opportunities  with  industry  as 
research  progresses  to  a  military  product. 
However,  the  finding  also  indicates  that 
more  than  30  percent  of  the  advanced 
development  technologies  are  still  good 
collaboration  candidates,  so  ample  oppor¬ 
tunities  for  the  Army  to  perform  research 
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with  industry  still  do  exist  for  advanced 
development  technologies. 

Discussion 


The  application  of  the  approach  shows 
its  value  as  a  management  tool.  In  a  time 
of  diminishing  resources,  being  able  to 
categorize  R&D  efforts  into  the  four  man¬ 
agement  domains  can  enable  resources  to 
be  saved  or  reallocated.  For  example, 
while  R&D  efforts  that  fall  into  the  lead 
category  must  be  funded  in  entirety,  those 
that  fall  into  the  initiate  or  participate  do¬ 
mains  can  be  cost-shared  with  industry; 
those  funds  can  either  be  saved  or  used 
elsewhere  to  fund  other  R&D  efforts.  In 
addition,  R&D  efforts  that  fall  into  the 
monitor  category  require  no  real  alloca¬ 
tion  of  resources  beyond  what  is  neces¬ 
sary  to  keep  abreast  of  the  industry;  thus, 
any  excess  resources  could  again  be  saved 
or  diverted  to  R&D  efforts  elsewhere  in 
the  organization. 

The  ability  to  categorize  R&D  efforts 
into  management  domains  also  has  some 
applicability  as  to  how  the  efforts  them¬ 
selves  are  managed.  While  efforts  in  the 
lead  domain  lend  themselves  to  standard 
contracts  involving  traditional  military 
suppliers,  those  in  the  initiate  and  partici¬ 
pate  domains  lend  themselves  to  collabo¬ 


rative  efforts  using  CAs  and  OTs.  There 
may  be  some  efficiencies  and  economies 
of  scale  to  managing  like  efforts  to¬ 
gether — for  example,  all  contract-based 
efforts  versus  all  OT-based  efforts. 

In  fact,  the  government  has  recognized 
that  there  are  potential  efficiencies  to  man¬ 
age  these  collaborative  efforts  together  and 
is  anticipating  a  substantial  increase  in  the 
use  of  collaborative  instruments  such  as 
CAs  and  OTs.  This  recognition  is  made 
apparent  by  recent  changes  implemented 
by  the  Defense  Contract  Management 
Command  (DCMC).  DCMC  has  desig¬ 
nated  four  regional  offices  to  administer 
the  Post  Award  Authority  (PAA)  of  OTs 
and  CAs  and  developed  specialized  ex¬ 
pertise  to  do  it.  This  new  service  is  being 
provided  to  both  the  Services  and  to  DoD 
agencies,  such  as  DARPA.  To  facilitate  the 
provision  of  this  new  service,  DCMC  will 
also  provide  limited  assistance  with  pre¬ 
award  negotiations  where  the  use  of  CAs 
and  OTs  is  being  considered. 

Our  study  shows  that  there  are  many 
technologies  where  the  Army  can  benefit 
by  performing  collaborative  R&D  with 
industry.  By  using  contractual  instruments 
such  as  CAs  and  OTs  and  by  taking  ad¬ 
vantage  of  the  new  services  being  offered 
by  DCMC,  the  DoD  has  the  tools  to  real¬ 
ize  the  benefits  offered  through  DoD-in- 
dustry  R&D  collaborations. 
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Endhotis 


1.  We  examined  a  number  of  alternatives 
in  our  search  for  a  tool.  Ultimately, 
we  adapted  the  concept  of  an  alterna¬ 
tive  that  was  developed  to  categorize 
Navy  technologies.  See  Saunders  et 
al.  (1995).  For  adaptation  details  on 
how  the  concept  was  adapted,  see 
Wong  (1998). 

2.  Defense  transition  from  full  funding 
and  control  of  R&D  to  collaborative 
R&D  with  industry  will  require  DoD 
organizations  to  examine  each  R&D 
technology  area  and  decide  whether 
it  might  be  a  candidate  for  collabora¬ 
tive  efforts.  The  matrix  described  here 
is  designed  as  a  tool  defense  organi¬ 
zations  can  use  to  gain  a  first-order 
cut  at  which  areas  might  be  worth  in¬ 
vestigating  as  collaboration  candi¬ 
dates.  The  matrix  is  not  designed  as  a 
decision  analysis  tool  to  make  defini¬ 
tive  R&D  project  funding  decisions. 
A  decision  on  whether  to  collaborate 
and  on  the  extent  of  defense  funding 
contributed  toward  any  collaborative 
effort  would,  of  course,  need  to  be 
made  on  a  project-by-project  basis  af¬ 
ter  considering  many  factors,  includ¬ 
ing  the  availability  and  suitability  of 
industry  partners,  the  precise  areas  of 
overlap,  benefits  to  the  government, 
security  considerations,  funding  re¬ 
quirements,  and  schedule  constraints. 


3.  We  envision  that  individual  DoD 
agencies  and  military  services  can  use 
the  matrix  tool  to  determine  which 
areas  are  collaboration  candidates  for 
a  particular  organization.  Conceiv¬ 
ably,  a  technology  area  that  is  a  good 
collaboration  candidate  for  one  DoD 
group  might  not  be  for  another  DoD 
group  because  of  different  focuses, 
priorities,  etc.  In  addition,  using  the 
tool  effectively  requires  that  the 
government  be  current  and  aware  of 
the  R&D  that  industry  is  performing 
in  the  technologies  being  considered 
for  collaboration.  This  requirement 
can  be  met  through  the  government 
experts  who  perform  the  “smart 
buyer”  function,  since  adequate 
performance  of  this  function  re¬ 
quires  both  in-depth  knowledge  and 
currency. 

4.  We  use  the  more  general  term  “re¬ 
sources”  to  include  cost  and  budget. 

5 .  The  domains  shown  in  the  figure  have 
fuzzy  borders.  There  is  no  line  or 
curve  on  the  framework  above,  below, 
left,  or  right  of  which  a  particular  man¬ 
agement  approach  can  be  judged  most 
appropriate. 
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6.  Government  funding  is  likely  to  be 
needed  for  technologies  in  the  lead  do¬ 
main  because  the  market  breadth  of 
these  technologies  is  limited  to  the 
government.  This  means  that,  for  the 
most  part,  commercial  applications  of 
the  technology  have  not  yet  been  rec¬ 
ognized.  Hence,  the  government 
might  not  be  able  to  find  industry  part¬ 
ners  to  collaborate  with  in  pursuing 
these  technologies. 

7.  Our  focus  here  was  on  basic  research 
(i.e.,  6. 1  activities);  the  other  two  S&T 
areas  are  exploratory  development 
and  advanced  development,  known  as 
6.2  and  6.3  activities,  respectively. 


8.  The  names  of  the  technologies  in  the 
box  are  the  names  used  in  the  R-l. 
Although  the  names  of  the  technolo¬ 
gies  are  generic,  our  placement  of  the 
technology  on  the  matrix  framework 
is  based  on  the  specific  R&D  activi¬ 
ties  that  were  funded  by  the  Army 
during  fiscal  year  1995.  That  is,  our 
placement  of  the  technology  does  not 
imply  that  all  research  that  might  fall 
under  the  generic  name  would  be  in 
the  domain  shown  in  our  Army  illus¬ 
tration. 

9.  The  matrix  described  earlier  was 
modified  to  reflect  the  Army. 
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LESSONS  LEARNED 


A  CASE  FOR  INTERNATIONAL 
COOPERATIVE  ACQUISITIONS 

LESSONS  FROM  DEVELOPING 
AND  EXECUTING  A  SECTION  27 
"QUAYLE"  AUTHORITY  PROGRAM 


Dr.  Alan  Childress  and  Lt  Col  James  Larsonf  USAF 


This  article  sets  out  international  cooperative  program  lessons  that  were  learned 
from  interviewing  29  past  and  present  stakeholders  from  the  United  Kingdom 
Ministry  of  Defence,  U.S.  industry,  U.S.  government  agencies,  and  joint  program 
office  personnel.  Not  surprisingly,  the  lessons  learned  suggest  that  devoting 
enormous  energy  and  focus  toward  understanding  each  other’s  frames  of 
reference  and  perspectives;  striving  to  work  together;  establishing  a  well- 
defined,  common  requirement  up  front;  and  continuous  senior-level  support 
are  factors  critical  to  success  in  an  international  cooperative  acquisition 
environment.  We  conclude  with  an  evaluation  of  the  program’s  organizational 
character. 


In  the  spirit  of  maintaining  past  Sec¬ 
retary  of  Defense  William  Perry’s 
strong  advocacy  for  developing  co¬ 
operative  acquisition  programs  with  our 
European  allies,  current  Secretary  of 
Defense  William  Cohen’s  March  1997 
policy  directive  states,  in  part,  that  at  the 
minimum  the  U.S.  military  must  “lever¬ 
age  U.S.  resources  through  cost  sharing 
and  economies  of  scale  afforded  by  inter¬ 
national  cooperative  research,  develop¬ 
ment,  production,  and  logistics  support 


programs.”  In  this  article  we  present  and 
discuss  several  lessons  learned  from  an  in¬ 
ternational  cooperative  acquisition — ini¬ 
tiated  in  1993 — that  largely  achieves  the 
objectives  of  Secretaries  Perry  and  Cohen. 

We  note  that  D’Agostino  (1996)  evalu¬ 
ated  and  compared  two  multinational 
weapons  development  efforts,  identifying 
multinational  political  and  management 
issues  that  exacerbated  technical  and 
schedule  problems.  She  described  risk  ar¬ 
eas  as  including: 
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•  number  of  countries  and  industries; 

•  differing  and  excessive  requirements; 

•  complex  cost  share  and  technical  work 
share  decisions; 

•  consortia  versus  prime  contractors ;  and 

•  international  program  office  staffing 
and  decision-making. 

Our  research,  more  focused  in  nature, 
complements,  amplifies,  and  adds  to  her 
conclusions  through  identifying  issues 
related  to  program  establishment  and  man¬ 
agement.  While  an  acknowledged 
D’Agostino  research  limitation  was  the 
lack  of  a  successful  program — she  based 
her  findings  on  a  canceled  program  and  a 
new  program — we  studied  an  ongoing 
program  that,  notwithstanding  schedule 
challenges,  appears  successful,  despite  the 
real  and  perceived  barriers  and  risks 
encountered. 

The  Program 


The  AN/AAQ-24  Directional  Infrared 
Countermeasures  (DIRCM)  program  is 
one  of  the  U.S.  Special  Operations 
Command’s  (USSOCOM’s)  highest  pri¬ 
ority  acquisition  programs.  This  urgently 
needed  aircraft  self-protection  suite  will 
provide  fast  and  accurate  threat  detection, 


processing,  tracking,  and  countermeasures 
to  defeat  current  and  future  generation 
infrared  missile  threats.  DIRCM  is  de¬ 
signed  for  installation  on  a  wide  range  of 
rotary  and  fixed-wing  aircraft.  For 
USSOCOM,  the  system  will  be  installed 
on  all  of  Air  Force  Special  Operations 
Command’s  (AFSOCS’s)  AC-130  gun- 
ships  and  MC-130  Combat  Talon  aircraft. 
Growth  to  counter  more  sophisticated 
threats  is  incorporated  into  the  program 
by  providing  a  path  that  allows  for  direct 
insertion  of  a  laser-based  countermeasure 
when  an  all-band  laser  is  developed.  These 
capabilities  made  the  DIRCM  system,  and 
others  like  it,  strong  candidates  during 
USSOCOM’s  initial  evaluation  of  the 
options  available. 

After  careful  consideration  of  the  alter¬ 
natives,  USSOCOM  initiated  the  DIRCM 
program  as  a  cooperative  acquisition  with 
the  United  Kingdom  Ministry  of  Defence 
(U.K.  MoD)  under  Section  27  of  the  Arms 
Export  Control  Act  (AECA)  (“Quayle” 
Authority).  Section  27  of  the  AECA  au¬ 
thorizes  the  Department  of  Defense  (DoD) 
to  enter  into  cooperative  projects  with  al¬ 
lies  and  friendly  countries  for  cooperative 
research,  development,  test,  and  evalua¬ 
tion  (RDT&E)  or  joint  production  (includ¬ 
ing  follow-on  support)  of  defense  articles, 
concurrent  production  of  a  defense  article 
that  was  jointly  developed  by  the  United 
States  and  allied  or  friendly  countries,  or 
U.S.  procurement  of  a  defense  article  or 
service  from  an  allied  or  friendly  country. 


Lt  Col  James  Larson,  USAF,  is  a  graduate  of  PMC  90-2,  Defense  Systems  Management  Col¬ 
lege.  He  is  currently  the  Directional  Infrared  Countermeasures  (DIRCM)  Deputy  Joint  Program 
Manager  (DJPM). 

Alan  Childress,  Ph.D.,  is  a  management  consultant  with  Booz-Allen  &  Hamilton. 
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“Quayle”  Authority  (Section  2350b)  en¬ 
ables  DoD  to  waive  certain  contracting 
and  procurement  requirements  in  carry¬ 
ing  out  contracts  under  a  Section  27 
cooperative  project. 

Prior  to  program  inception,  the  two 
countries’  procurement  and  legal  staffs 
developed  and  negotiated  an  acceptable 
Memorandum  of  Understanding  (MOU). 
Given  the  program’s  urgency  and  a  strong 
desire  on  the  part  of  the  participants  to 
establish  a  firm  foundation  for  the 
program’s  success,  both  staffs  felt  the  best 
means  to  keep  the  negotiations  on  track 
was  to  leave  out  politically  charged  items 
such  as  cost  and  work  share  arrangements. 
Within  the  framework  of  the  “Quayle” 
Authority,  the  DIRCM  MOU  allows  the 
U.K.  MoD  to  competitively  award  a  con¬ 
tract  on  behalf  of  USSOCOM.  The  U.K. 
MoD  owns  and  manages  the  contract  with 
the  DIRCM  prime  contractor,  Northrop 
Grumman  Electronics  and  Systems 
Integration  International,  Inc.,  (NGESII) 
Rolling  Meadows,  IL. 

The  DIRCM  program  is  unusual  in  that 
it  is  one  of  the  first  cooperative  develop¬ 
ment  and  production  projects  undertaken 
by  a  U.S.  agency  wherein  the  allied  coun¬ 
try  owns  the  contract  with  industry.  In 
addition,  it  may  be  the  first  program  where 
the  U.K.  MoD  has  led  a  collaborative  pro¬ 
curement  with  the  United  States  in  which 
the  prime  contractor  is  one  of  the  major 
U.S.  defense  contractors. 

Total  U.S.  programmatic  cost  savings, 
documented  in  the  program’s  1996  David 
Packard  Acquisition  Excellence  Award 
narrative,  amount  to  $80  million. 


Overview  of  DIRCM 
Program  Management 


As  noted,  the  U.K.  MoD  owns  and 
manages  the  DIRCM  contract,  currently 
valued  at  over  $400  million  for  joint  U.K./ 
USSOCOM  content  as  well  as  United 
Kingdom-  and  USSOCOM-unique  re¬ 
quirements.  The  contract  is  to  develop, 
produce,  install,  field,  and  sustain  approxi¬ 
mately  131  DIRCM  systems  on  the  U.K. 
fixed-  and  rotary-wing  fleet  and  59  sys¬ 
tems  on  the  AFSOC  AC/MC-130  fleet. 


at 


both  sfaffffs  felt 
the  best  means  to 
keep  the  negotia¬ 
tions  bk  track  was 
to  leave  out  politi¬ 
cally  charged  items 
such  as  cost  and 
work  share 
arrangements." 


The  fixed-price 
(FP)  basic  con¬ 
tract,  awarded 
under  a  total  sys¬ 
tems  perfor¬ 
mance  responsi- 
bility  (TSPR) 
philosophy,  is 
for  the  joint 
engineering, 
manufacturing, 
and  develop¬ 
ment  (EMD)  phase  and  U.K.  production 
and  sustainment  phases,  and  includes 
priced  options  for  USSOCOM’s  produc¬ 
tion  and  sustainment  phases.  The  MOU 
to  enter  into  a  cooperative  program  be¬ 
tween  the  United  States  and  the  United 
Kingdom  was  signed  in  June  1994  and  the 
EMD  contract  with  Northrop  Grumman 
was  signed  in  March  1995. 

The  DIRCM  program  manager  is  a 
U.K.  Ministry  of  Defence  (MoD)  civilian. 
There  are  U.S.  and  U.K.  joint  program 
offices  (JPOs),  with  each  office  headed  by 
a  deputy  joint  program  manager  (DJPM). 


The  USSOCOM  JPO  at  MacDill  Air 
Force  Base,  FL,  is  staffed  by  a  handful  of 
military  and  civilian  managers,  augmented 
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by  a  team  of  contractor  technical  support 
personnel.  The  MoD  JPO  in  Bristol,  En¬ 
gland,  is  staffed  by  several  full-time  U.K. 
civil  servant  managers  and  one 
USSOCOM  civil  servant,  augmented  by 
off-site  specialized  engineering  support. 
The  U.K.  program  manager  is  co-located 
with  his  U.K.  deputy  in  Bristol.  In  addi¬ 
tion,  the  United  Kingdom  has  placed  an 
Integrated  Logistics  Support  (ILS)  man¬ 
ager  on-site  at  the  prime  contractor.  A 
steering  committee  comprises  U.K.  and 
U.S.  acquisition  executives  (Figure  1). 

In  addition  to  providing  functional  (en¬ 
gineering,  test,  ELS,  software,  etc.)  con¬ 
sultation  to  the  U.K.  program  manager, 
USSOCOM  is  responsible  for  managing 
program-wide  developmental  testing  at 


U.S.  test  facilities  such  as  the  Air  Force 
Electronic  Warfare  Evaluation  Simulator 
(AFEWES),  Army  Research  Laboratory 
(ARL),  Eglin  Air  Force  Base  test  ranges, 
and  the  White  Sands  Missile  Range  Aerial 
Cable  Facility.  USSOCOM  also  assists  in 
the  execution  of  that  portion  of  the  con¬ 
tract  to  outfit  the  Air  Force  Special  Op¬ 
erations  Command  (AFSOC)  AC/MC- 
130  fleet  with  DIRCM  systems.  The 
USSOCOM  JPO  is  managed  through  a 
two-tier  integrated  product  team  (IPT) 
structure,  with  U.K.  and  prime  contractor 
representation  in  the  upper  tier.  These  IPTs 
draw  extensively  on  Service  and  OSD 
expertise  in  the  areas  of  engineering,  test, 
logistics,  and  aircraft  integration.  Pres¬ 
ently  the  program  is  in  the  latter  stages  of 
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EMD,  with  production  scheduled  to  start 
in  1998. 


Data  Sources  and  Research  Purpose 

The  sources  of  research  data  for  this 
study  were  program  documentation  and 
individual  interviews  with  29  key  decision 
makers,  stakeholders,  managers,  and  func¬ 
tional  experts  substantially  affecting  the 
DIRCM  program,  past  and  present.  In 
addition  to  a  semi-structured  interview 
format  designed  to  gather  programmatic 
and  technical  data,  interviewees  were 
asked  what  lessons  they  learned  (and  re¬ 
lated  observations)  from  their  involvement 
in  the  DIRCM  program.  We  generally 
conducted  the  interviews  in  the  home  of¬ 
fice  of  interviewees  or  in  a  neutral  setting. 
The  research  purpose  was  to  document 
DIRCM’s  experience  in  the  form  of  les¬ 
sons  learned.  In  particular,  we  wished  to 
share  the  program’s  Section  27  “Quayle” 
Authority  successes  and  shortcomings 
with  future  international  cooperative 
programs. 

Lessons  Learned 


Summarized  Section  27  and  related  les¬ 
sons  learned,  and  the  associated  rationale 
for  these  experiences,  are  categorized  as 
follows: 

•  drafting  and  negotiating  the  MOU; 

•  defining  the  requirement; 

•  cross-cultural  communications  and 
teamwork; 


•  personalities,  professional  skill  sets, 
and  motivation; 

•  writing,  negotiating,  and  executing  the 
contract; 

•  TSPR/FP-type  contracting; 

•  management  continuity ; 

•  contractor  program  management  sup¬ 
port; 

•  IPT  management;  and 

•  leadership. 

Drafting  and  Negotiating  the  MOU 

Lesson:  When  contemplating  a  Section 
27  program,  seek  Service  headquarters  or 
OSD  assistance  and  sponsorship. 

Rationale:  The  first  USSOCOM 
DIRCM  program  manager  learned  from 
OSD  officials  at  a  very  early  stage  that  if 
he  worked  the  MOU  procedures  himself 
through  USSOCOM  channels,  the  MOU 
process  could  take  18-24  months  rather 
than  the  6-8  months  he  could  afford  for 
AFSOC’s  urgent  requirement.  No  Section 
27  “Quayle”  Authority  (the  allied  coun¬ 
try  is  the  procurement  agent)  codevelop¬ 
ment,  coproduction  procurement  had 
been  seriously  contemplated  before; 
USSOCOM  would  have  had  to  develop 
the  documentation  internally  and  staff  it 
through  the  Joint  Staff  and  the  cognizant 
Services.  He  asked  a  key  OSD  mid-level 
acquisition  official  for  help  and  received 
it.  The  OSD  official  and  the  U.K.  program 
manager  said  that  direct  senior  executive 
involvement  would  be  very  helpful  in  ex- 
pediting  the  MOU  process.  The 
USSOCOM  acquisition  executive  (AE), 
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an  innovator  in  tailored  acquisition,  made 
supporting  calls  for  cooperation  to  OSD 
and  U.K.  MoD  acquisition  executives. 
This  direct  high-level  support  from  the 
beginning  also  sent  a  powerful  message 
to  the  United  Kingdom  that  USSOCOM 
was  seriously  interested  in  a  cooperative 
effort. 


Lesson:  Enlist  experienced  negotiators 
to  help  negotiate  the  MOU. 

Rationale:  Allied  countries  have  pro¬ 
fessional  negotiators  who  consistently 
outperform  less  experienced  U.S.  negoti¬ 
ating  teams.  The  Defense  Security  Assis¬ 
tance  Agency  (DS AA)  General  Counsel, 
who  helped  negotiate  the  DIRCM  MOU, 
suggests  that  the  U.S.  defense  establish¬ 
ment  should  not  “send  a  boy  out  to  do  a 
man’s  work.  [We]  need  to  send  out  on  a 
negotiating  team  experienced  people  who 
‘know  how  to/been  there/done  that.’”  The 


skilled  OSD-led  negotiating  team  was 
very  successful  in  achieving  USSOCOM’s 
objectives.  Equally  significant,  the  team 
completed  the  negotiations  in  the  very 

short  period  of  a 


"Allied  countries  :  weekian,d  Wlth 
hove  professional  very  little  acn- 
negotiators  who  mony  (which 

consistently  out-  could  have  seri- 
perform  less  ously  jeopar- 

experienced  U.S.  dized  long-term 

negotiating  teams."  relationships  on 

the  program). 
(Note:  The  au¬ 
thors  recognize  DSMC’s  international 
management  training  courses.  One,  Alan 
Childress,  is  a  graduate  of  the  Advanced 
International  Management  Workshop  and 
highly  recommends  that  training  to  anyone 
contemplating  international  cooperative 
acquisitions.) 


Defining  the  Requirement 

Lesson:  A  well-defined,  focused  re¬ 
quirement  that  includes  a  commonality  of 
interest  is  essential  for  success.  Both  sides 
must  strongly  desire  to  do  the  same  thing. 

Rationale:  In  the  early  stages  of  form¬ 
ing  the  partnership,  the  U.S.  and  U.K.  pro¬ 
gram  managers  encountered  several  ob¬ 
stacles,  some  caused  by  cultural  differ¬ 
ences  and  baggage  from  earlier  unrelated 
attempts  at  cooperative  efforts  that  had 
failed.  These  obstacles  could  have  easily 
threatened  collaboration.  The  significant 
savings  in  time  and  money  was  very  im¬ 
portant  to  collaborating;  however,  the  goal 
of  defeating  a  similar  list  of  threats  under 
like  scenarios  of  operation  was  the  com¬ 
mon  thread  that  secured,  and  continues  to 
secure,  the  partnership.  The  interviewees 
advise  future  program  managers  to  ac¬ 
knowledge  and  value  differences  while 
working  hard  toward  mutually  beneficial 
solutions  and  avoiding  compromises  that 
dilute  the  objectives  of  one  of  more  of  the 
parties. 

Lesson:  Section  27  works  best  if,  in 
addition  to  a  common  requirement,  the 
partnership  is  formed  from  the  bottom  up. 

Rationale:  Early  in  the  program,  the 
U.S.  DIRCM  managers  discovered  there 
were  common  U.S.  and  U.K.  requirements 
they  could  merge  for  joint  execution.  Both 
countries  were  in  formal  stages  of  going 
forward  with  similar  needs.  When  U.S. 
program-level  officers  approached  the 
U.K.  program  manager  and  his  deputy 
with  congruent  requirements,  as  well  as 
resources,  they  saw  that  a  good  marriage 
was  possible.  As  a  result,  combining  the 
efforts  was  approved  all  the  way  up  the 
chain.  In  contrasting  cases,  according  to 
former  DSAA  General  Counsel  Susan 
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Ludlow-MacMurray,  both  countries  al¬ 
ready  had  their  own  programs  ongoing 
when  they  were  directed  from  the  top, 
politically,  to  merge,  which  caused  dilu¬ 
tion  of  authority  and  responsibility  and 
dissatisfaction  in  one  or  both  sides’  man¬ 
agement.  She  suggests  that  international 
programs  driven  by  bottom-up  motivation 
generally  succeed.  Those  programs  that 
emanate  from  the  top  down  (Service  Sec¬ 
retary  or  OSD  level)  generally  do  not  suc¬ 
ceed;  they  die  from  lack  of  a  mid-level 
buy-in  or  sponsorship. 

Cross-Cultural  Communications 
and  Teamwork 

Lesson:  Approach  a  potential  interna¬ 
tional  cooperative  acquisition  partner  with 
a  very  small  team  of  highly  skilled  people 
and  plan  to  agree  to  limit  the  number  of 
U.S.  program  staff  participants  directly 
involved  in  program  startup  and  execu¬ 
tion. 

Rationale:  The  British  were  apprehen¬ 
sive  that  a  large,  Service-level  program 
would  attempt  to  subsume  their  ongoing 
program  once  a  cooperative  agreement 
was  in  place.  They  made  this  point  clearly 
and  pointed  out  past  examples  of  failed 
efforts  when  approached  by  the  U.S.  team. 
Their  primary  concern  was  that  of  losing 
control  and  the  focus  of  meetings  if  at¬ 
tended  by  a  large  U.S.  contingent.  They 
clearly  stated  they  would  pull  out  of  an 
agreement  if  the  United  States  attempted 
to  modify  their  schedule  or  could  not  ac¬ 
cept  junior  partner  status.  After  a  few 
meetings,  they  saw  that  the  USSOCOM 
organization  was  relatively  small  and  ag¬ 
ile  and,  like  them,  embraced  acquisition 
streamlining. 


Lesson:  Each  side  in  an  international 
cooperative  program  must  dedicate  an 
enormous  effort  to  understand  the  culture, 
motivations,  and  idiosyncrasies  of  the 
people  and  bureaucracies  of  the  other 
country. 

Rationale:  U.K.  interviewees  empha¬ 
sized  that  realizing  the  magnitude  of  cul¬ 
tural  differences  was  quite  a  shock.  The 
first  U.K.  DJPM  said  working  together  on 
this  program  illustrates  the  true  meaning 
of  the  concept  of  “two  countries  divided 
by  a  common  language.”  He  recommends 
taking  a  gloves-off  approach  and  telling 
each  other  clearly  and  openly  how  issues 
are  being  viewed,  or  be  prepared  to  suffer 
the  consequence  of  miscommunication. 
Two  U.K.  interviewees  stated  that  com¬ 


mitment  from 
senior  manage¬ 
ment  on  both 
sides  is  an  abso¬ 
lute  require¬ 
ment  to  allow 
enough  inter¬ 
change  between 
the  people  do¬ 
ing  the  job. 
They  are  not 
sure  there  had 


"Each  side  in  an ' ' "  ;|;v 
international  coop¬ 
erative  program 
must  dedicate  an 
enormous  effort  to 
understand  the  ;; 
culture,  motivations, 
and  idiosyncrasies 
of  the  people  and 
bureaucracies  of  the 
other  country." 


been  sufficient 


senior  management  emphasis  on  the 
DIRCM  project  to  achieve  the  level  of  co¬ 
operation  that  might  have  been.  In  particu¬ 
lar,  they  feel  senior  managers  might  have 
committed  more  travel  resources  to  allow 


this  interchange. 

U.S.  interviewees  suggest  that  a  U.S. 
DJPM  must,  at  times,  look  at  things 
through  the  eyes  of  his  or  her  counterpart 
to  understand  the  other’s  point  of  view. 
For  example,  a  U.S.  DJPM  had  a  difficult 
time  agreeing  with  his  counterpart  on  an 
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accurate  assessment  of  the  program’s 
schedule.  In  his  eyes  the  program  had 
slipped  a  considerable  amount,  on  the  or¬ 
der  of  12  months.  The  U.K.  DJPM  main¬ 
tained  just  as  strongly  that  the  program  had 
hardly  slipped  at  all,  maybe  one  or  two 
months.  After  having  this  disagreement  in 

front  of  the  re- 
"IWo  British  spective  acqui- 

interviewees  sition  execu- 

commented  on  the  tives,  the  U.S. 

synergism  realized  DJPM  came  to 

from  international  realize  the  Brit- 

cooperation."  ish  typically 

measured  pro¬ 
grams  in  relation  to  the  end  date  of  the 
contract,  while  the  United  States  typically 
uses  initial  operational  capability,  or  when 
the  system  first  makes  it  to  the  field.  In 
the  U.S.  case,  the  initial  operational  capa¬ 
bility  had  slipped  12  months,  but  the  con¬ 
tractor  was  able  to  adjust  production  and 
installation  scenarios  to  maintain  the  same 
contract  end  date.  Essentially,  both  DJPMs 
were  right,  locked  in  violent  agreement. 
They  just  did  not  know  it. 

Lesson:  When  assigning  functional  ex¬ 
pertise,  U.S.  program  managers  should 
strive  to  achieve  a  synergistic  balance  with 
other  participants’  team  members.  By 
drawing  on  key  areas  of  expertise  from 
each  country  while  trying  to  avoid  too 
much  overlap  (and  high  potential  for  per¬ 
sonal  competition  and  conflict),  the  over¬ 
all  team  will  be  more  effective  and  agile. 

Rationale:  Two  British  interviewees 
commented  on  the  synergism  realized 
from  international  cooperation.  One  sug¬ 
gested  that  while  either  side  would  have 
done  a  grand  job  on  its  own,  “the  fact  that 
(the  technicians)  know  how  to  bounce 
things  off  each  other  has  been  a  great 


benefit;  we  should  keep  that  well  in  our 
sights. ..and  on  the  management  side  there 
are  differences  in  approach,  which,  pooled 
together,  benefit  both  parties.”  DIRCM’s 
MoD  executive,  John  Allen,  noted  that  “no 
doubt  USSOCOM  has  a  better  knowledge 
of  both  U.S.  industry  generally  and 
Northrop  in  particular.  Both  sets  of  expe¬ 
rience  brought  to  manage  one  particular 
contract  is  working  well. ..Northrop 
Grumman  knows  that  USSOCOM  is  a 
better-informed  customer  than  we  are.. .we 
can  draw  from  that  experience.”  A 
Northrop  senior  manager  commented  that 
the  integration  of  U.K.,  USSOCOM,  and 
Northrop  technical  specialists  “has  been 
outstanding... benefits  to  the  United  King¬ 
dom  and  United  States  in  operating  that 
way  are  tremendous.  I  can’t  over-stress 
that.”  DIRCM’s  program  manager  argues 
that  from  the  collaboration  he  is  “abso¬ 
lutely  committed  that  we  are  both  getting 
a  better  product  out  of  this.” 

Personalities,  Professional  Skill  Sets, 
and  Motivation 

Lesson:  When  contemplating  the  for¬ 
mation  of  an  acquisition  partnership  with 
a  potential  international  partner,  U.S. 
agencies  should  recruit  or  place  their 
most  technically  competent,  strongest 
personalities  in  the  initial  contact  and 
management  teams. 

Rationale:  U.K.  program  officers  re¬ 
marked  that  they  were  very  impressed, 
particularly  in  the  early  stages  of  forming 
the  DIRCM  partnership,  by  the  personali¬ 
ties,  drive,  and  desire  to  succeed  of  the 
founding  U.S.  program  team  members. 
They  suggest  the  marriage  probably  would 
not  have  happened  without  the  intense 
interest  of  an  OSD  supporting  official,  or 
the  doggedness  of  the  first  DJPM  to  “make 
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it  happen.”  The  British  look  hard  at  per¬ 
sonalities  when  contemplating  a  busi¬ 
ness  relationship.  The  USSOCOM  AE, 
Gary  Smith,  was  seen  as  an  acquisition 
innovator  with  whom  they  could  do 
business. 

Lesson:  U.S.  decision  makers  should 
implement  a  deliberate  personnel  policy 
to  hire  or  place  and  retain  the  best  pro¬ 
gram  management  and  technical  skill  sets 
available  for  Section  27  programs.  In  ad¬ 
dition,  executives  should  attempt  to  recruit 
personnel  with  international  cooperative 
acquisition  training  and  experience  (see 
Lesson  1,  Drafting  the  MOU). 

Rationale:  The  U.S.  AE’s  placement  of 
management  personalities  and  overlap¬ 
ping  skill  sets  is  a  positive  lesson.  The  first 
DJPM  was  a  contracting  officer,  acquisi¬ 
tion  professional,  and  operator.  The  sec¬ 
ond  DJPM’s  background  was  operations 
and  acquisition.  The  current  DJPM  is  an 
acquisition  professional,  while  the  support 
contractor  technical  director  has  an  opera¬ 
tional  and  acquisition  professional  back¬ 
ground  with  experience  as  a  program  man¬ 
ager  in  industry.  First-rate  technical  pro¬ 
fessionals  were  hired  to  support  these 
managers.  The  U.K.  program  manager 
professed  he  is  understanding  of  the  fact 
that  U.S.  personnel  must  learn  his  way  of 
doing  business  as  well  as  sustaining  U.S. 
policies  and  procedures. 

Lesson:  When  entering  into  an  acqui¬ 
sition  partnership  where  the  other  coun¬ 
try  owns  the  contract  with  industry,  the 
U.S.  side  must  be  prepared  to  accept  a 
subordinate  management  role. 

Rationale:  U.K.  and  Northrop 
Grumman  officials  point  out  the  positive 
effect  on  the  relationship  resulting  from 


tiation  process, 
they  worked 
with  the  U.K. 
program  man¬ 
ager  and  did  not 
try  to  lead  him. 


USSOCOM  team  recognition,  from  the 
outset,  that  DIRCM  would  be  a  U.K.  con¬ 
tract.  The  first  U.K.  program  manager 
made  it  clear  early  in  the  project  that  he 
was  the  program  manager,  that  he  called 
the  shots,  and  that  the  contract  terms  were 
United  Kingdom  terms,  not  those  of  the 
United  States.  When  the  first  and  second 
U.S.  DJPMs  were  in  London  during  the 
contract  nego- 

##The  first  U.K. 
program  manager 
made  it  clear  early 
in  the  prefect  that  he 
was  the  program 
manager,  that  he 

A  good  illustra-  «"*■««*  *he  shots,  and 

don  of  this  les-  *hut  #he  con#r«t  . 
i  ,  ,  terms  were  United 

w  ’  y  Kingdom  terms,  no* 
Northrop  s  con-  fhose  ef|||e  ynl#e(I 

tracting  director,  states." 

He  was  attend¬ 
ing  a  briefing  by 

the  U.K.  program  manager,  sitting  behind 
the  overlapping  U.S.  DJPMs  during  their 
changeover  phase.  He  said  the  second 
U.S.  DJPM  (then-Lt  Col  Karl  “Chip” 
Kochel)  turned  to  his  predecessor  (Lt  Col 
Jim  Pennock)  and  asked  a  question. 
Pennock’s  reply,  as  he  pointed  to  the  U.K. 
program  manager,  was  “Ask  your  program 
manager.” 


Writing,  Negotiating,  and 
Executing  the  Contract 

Lesson:  When  negotiating  an  allied-led 
RDT&E  TSPR/FP  contract  with  U.S.- 
based  defense  firms,  the  program 
manager,  with  the  U.S.  Deputy  DJPM, 
should  meticulously  precoordinate  the 
developmental  and  operational  testing 
terms,  conditions,  and  standards  with 
the  appropriate  U.S.  test  agencies. 
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Rationale:  The  U.K.-owned  EMD  con¬ 
tract  with  Northrop  Grumman  gave 
Northrop  total  systems  performance  re¬ 
sponsibility,  including  developmental  test¬ 
ing.  The  testing  program  included  the  use 
of  U.S.  Air  Force  test  activities.  Differ¬ 
ences  in  test  standards,  procedures,  and 
philosophy  emerged  after  development 

was  under  way 

^ln  cooperative 
acquisitions,  certain 
elements  of  the 
requirement  may 
be  unique  to  each 
country." 


and  have  con¬ 
tinued  to  plague 
the  program 
manager.  U.K. 
interviewees, 
while  admitting 
that  DIRCM 


technology  presents  unanticipated  test 
challenges,  argue  that  there  has  been  a  ten¬ 
dency  from  the  U.S.  test  community  to  try 
to  run  testing  as  if  it  were  a  cost-plus  type 
development  contract,  when  in  fact  it  is  a 
fixed-price  contract.  Two  Northrop 
Grumman  officials  suggest  that  the  pro¬ 
gram  manager,  with  program  goals  in 
mind,  should  have  the  final  word  regard¬ 
ing  testing.  It  has  not  worked  out  in  that 
manner,  causing  confusion  at  times. 


Lesson:  When  developing  the  Section 
27  contract  with  industry,  write  a  U.S.- 
only  portion  of  the  contract  to  help  in  ob¬ 
taining  support-system  information;  tie 
U.S.  payments  to  contract  data  require¬ 
ments  list  (CDRL)  deliveries. 

Rationale:  In  cooperative  acquisitions, 
certain  elements  of  the  requirement  may 
be  unique  to  each  country.  For  example, 
U.S.  logisticians  require  product  and 
contract  information  to  establish  a  cost- 
effective  support  infrastructure.  Also, 
the  “system”  requires  them  to  have  a  con¬ 
tract  number  in  the  U.S. -contract-number 
format  since  their  software  does  not 


accommodate  the  contract  number  format 
used  by  some  other  nations.  In  addition, 
contract  specifications  are  generally  more 
unbounded,  causing  U.S.  JPO  logisticians 
a  small  problem  managing  compliance, 
particularly  with  regard  to  the  CDRLs 
(contract  deliverables).  The  delivery  of 
data,  in  some  cases,  is  more  important  than 
the  product  itself. 

TSPR/FP-type  Contracting 

Lesson:  When  faced  with  total  systems 
performance  responsibility/fixed  price 
(TSPR/FP)-type  contracting  in  a  coopera¬ 
tive  acquisition,  U.S.  DJPMs  and  U.S. 
contractors  should  take  care  to  fully  un¬ 
derstand  the  concept  of  TSPR/FP  contract¬ 
ing  and  the  pitfalls  of  execution  in  the  U.S. 
acquisition  environment. 

Rationale:  TSPR/FP  contracting  is 
generally  not  alien  to  the  U.S.  acquisition 
culture;  however,  for  various  reasons  U.S. 
agencies  tend  to  drift  away  from  imple¬ 
menting  true  TSPR,  especially  during 
times  of  technical  challenges.  At  times  the 
temptation  for  U.S.  program  managers  and 
their  functional  team  members  is  too  great 
to  resist  getting  directly  involved  in  “help¬ 
ing”  the  contractor  work  through  the  prob¬ 
lems.  However,  this  approach  typically 
ends  up  with  the  government  performing 
work  or  functions  that  the  prime  contrac¬ 
tor  was  paid  to  do  while  at  the  same  time 
possibly  absolving  the  contractor  of 
responsibly  for  failing  to  perform. 

While  the  United  Kingdom  fully  sup¬ 
ports  and  accepts  Northrop  Grumman  re¬ 
sponsibility  and  judgment  on  the  require¬ 
ments,  the  U.S.  JPO  and  Northrop  are  ex¬ 
periencing  problems  with  U.S.  inspection 
and  test  agencies  acceptance  of  TSPR. 
For  example,  a  Northrop  Grumman  in¬ 
terviewee  complains  that  the  Defense 
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Contracts  Auditing  Agency  is  inspecting 
their  Group  A  component  installation  on 
an  incremental  basis  as  they  deem  fit,  but 
Northrop  does  not  have  “incremental  ab¬ 
solution”  to  go  along  with  the  inspection. 
Northrop  still  has,  at  the  end  of  the  pro¬ 
gram  or  at  the  end  of  the  modification,  the 
obligation  to  present  it  to  the  government 
and  the  government  has  the  right  to  ac¬ 
cept  or  reject,  even  though  they  agreed  on 
something  in  the  normal  course  of  doing 
the  modification. 

An  interesting  compromise  between 
U.K.  and  U.S.  contract  management  ap¬ 
proaches  has  emerged  in  the  DIRCM  pro¬ 
gram.  The  U.S.  side  introduced  a  review 
process  to  TSPR  contracting.  The  contract 
is  being  executed  using  this  review  pro¬ 
cess  to  help  the  contractor,  but  the  gov¬ 
ernment  does  not  sign  off  on  formal  re¬ 
view  documents.  In  the  words  of  a  DJPM, 
“Northrop  performs  on  the  contract  and 
we  oversee  their  performance.  If  they  con¬ 
vince  us  that  it’s  great  and  they  are  ready 
for  preliminary  design  review  (PDR),  we 
complete  that  event;  if  they  convince  us 
they  are  ready  for  critical  design  review 
(CDR),  then  that  is  fine.  Or  if  they  are  not 
ready  in  any  review,  they  don’t  go  into  the 
next  event  until  it’s  satisfactory.”  While 
the  review  process  is  not  contractual,  it 
appears  to  strike  a  balance  between  the 
U.S.  cost-plus  “stay  in  their  knickers”  ap¬ 
proach  and  the  “hands-off’  British  TSPR 
system.  In  the  DJPM’s  view,  it  puts  the 
government  in  a  better  position  to  help  the 
contractor  work  through  problems  in  early 
stages,  “If  you  wait,  you  are  going  to  cre¬ 
ate  a  wave  that  you  can’t  overcome.”  Fi¬ 
nally,  according  to  the  present  U.S .  DJPM, 
TSPR/FP  contracting,  combined  with  IPT 
management,  is  saving  enormous  program 
office  resource  costs. 


Management  Continuity 


Lesson;  The  complexity  of  managing 
Section  27  codevelopment  programs, 
along  with  the  benefits  of  preserving  es¬ 
tablished  international  personal  relation¬ 
ships,  requires  that  decision  makers  estab¬ 
lish  and  maintain  a  management  structure 
that  provides  management  continuity  and 
overlaps  systemic  personnel  rotation. 

Rationale:  When  the  first  U.S.  DJPM 
was  notified  of  his  reassignment,  the  U.K. 
team  was  worried  about  the  impact  of  los¬ 
ing  such  a  strong  player  on  short  notice. 
They  stated  their  concerns,  and  were  re¬ 
lieved  knowing  that  the  contractor  tech¬ 
nical  director  would  be  the  “glue”  person 
to  hold  the  U.S.  side  together  during  the 
transition.  They 


claim  his  tech¬ 
nical  expertise 
and  personality 
were  critical  in 
the  U.S.  DJPM 
transitions.  The 
founding  U.K. 
program  man¬ 
ager  stated  that, 


"Finally,  according 
to  the  present  !!•$• 
DJPM,  TSPR/FP 
contracting,  coni- 
bined  with  IPT  man* 
agement,  is  saving 
enormous  program 
office  resource 
costs/' 


“The  contractor 

management  support  key  players  have 
been  providing  the  continuity  that  the 
majors  and  lieutenant  colonels  haven’t 
been  able  to  provide.  That  was  important. 
It  couldn’t  have  happened  without  that 
continuity.”  According  to  him,  the  British 
have  an  ambition  that  their  program  man¬ 
agers  should  stay  about  three  or  four  years 
minimum. 

This  concern  relates  to  contractor  as 
well  as  military  management.  Regarding 
Northrop’s  management  turnover,  the  se¬ 
nior  U.K.  interviewee  stated  that  the 
present  management  is  very  good,  but 
“nevertheless  the  continuing  change  of 
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personalities  within  the  company  is  not 
very  reassuring  and  we  continually  have 
to  watch  that  ” 

Contractor  Program 
Management  Support 

Lesson:  When  structuring  and  main¬ 
taining  an  international  program  office, 
contractor  management  support  provides 
the  program  manager  choices  in  tailoring, 
flexibility,  and  continuity  not  available  in 
an  all-government  solution. 

Rationale:  Contractor-provided  man¬ 
agement  support  was  vital  to  forming  and 
maintaining  the  U.S.  JPO;  the  government 
billets  and  technical  expertise  were  not 
available  during  program  formation.  The 
USSOCOM  AE  commented  that  “we  used 

to  have  a  very 
"The  USSOCOM  AE  large  laboratory 

commented  that  "we  structure  that 

used  to  have  a  very  provided  us  in- 

large  laboratory  house  expertise, 

structure  that  pro-  but  that’s  going 

vided  us  in-house  away.  We  have 

expertise,  but  that's  ;  t0  hire  contrac_ 

going  away."  tors  that  have 

the  in-depth  ex¬ 
pertise.  It’s  sensible  to  hire  technical  sup¬ 
port  in  today’s  downsizing  environment.” 
The  first  DJPM  relied  on  Systems  Engi¬ 
neering  and  Technical  Assistance  (SETA) 
support  because,  “You  need  a  core  of  ci¬ 
vilians  that  can  maintain  continuity  across 
the  program.”  The  present  DJPM  main¬ 
tains  that  through  contractor-provided 
management  support:  Taxpayers  get  a 
break  in  that  programs  do  not  have  to  carry 
inappropriate  personnel;  the  contractor 
selection  process  tends  to  attract  and  main¬ 
tain  the  best  people;  and  the  JPO  tends  to 
exhibit  greater  teamwork  among  func¬ 
tional  areas  through  contractor  personnel. 


IPT  Management 

Lesson:  A  controlled  IPT  process  is 
essential  to  effectively  managing  a  com¬ 
plex  and  geographically  scattered  Section 
27  program. 

Rationale:  Before  implementing  IPT 
management,  the  U.S.  DJPM  was  having 
significant  problems  integrating  and  con¬ 
trolling  his  extended  acquisition  organi¬ 
zation.  The  program  office  was  small,  with 
support  staff  and  stakeholder  organiza¬ 
tions  scattered  globally.  Support  organi¬ 
zation  staff  were  communicating  and  de¬ 
ciding  among  themselves,  without  includ¬ 
ing  the  JPO  in  the  loop.  The  IPT  process 
sharply  reduced  those  control  problems 
and  brought  a  spirit  of  teamwork  and  ac¬ 
complishment  to  the  DIRCM  program. 
Also,  according  to  a  U.K.  interviewee, 
“Cutting  down  meeting  participation 
(through  the  USSOCOM’s  use  of  IPTs 
outside  of  normal  joint  U.S./U.K.  meet¬ 
ings)  helps  because  one  of  the  problems 
that  we  had,  certainly  early  on,  was  at  each 
meeting  there  would  be  somebody  new 
who  really  knew  very  little  about  the 
program  and  what  had  gone  on  in  the 
past.  And  they’d  start  asking  the  ques¬ 
tions  that  were  addressed  2  or  3  months 
ago.  It  was  always  as  if  we  had  to  bring 
them  up  to  speed  before  the  meeting  could 
proceed.” 

A  DJPM  interviewee  made  these 
recommendations  for  creating  IPTs: 

•  Ensure  top-level  support  through  re¬ 
view  and  signature  on  the  IPT  charter. 

•  Get  the  right  disciplines  and  individu¬ 
als  on  the  team;  include  industry.  As 
the  IPT  leader,  the  DJPM  manages 
membership  and  participation. 
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•  Manage  the  meetings  and  keep  them 
focused.  Do  not  allow  old  issues  to  be 
reopened  if  they  have  already  been 
closed  by  mutual  agreement. 

•  Keep  the  meetings  on  track  to  preclude 
the  waste  of  time. 

•  Only  address  the  highlights  of  each 
functional  area/topic. 

•  Encourage  the  functional  representa¬ 
tives  to  create  their  own  mini  sub-IPTs, 
working  the  details  in  them  rather  than 
bringing  detailed  technical  issues  to  the 
overall  DPT  forum. 

Leadership 

Lesson:  Involved  and  decisive  leader¬ 
ship,  from  the  top  down,  is  particularly 
essential  to  a  Section  27  program. 

Rationale:  In  the  early  days  of  form¬ 
ing  the  partnership  with  the  United  King¬ 
dom,  indecisiveness  on  the  part  of  the  U.S. 
leadership  would  most  likely  have  resulted 
in  failure.  An  industry  interviewee  ob¬ 
served  this  of  the  program,  “A  key  ele¬ 
ment  in  making  the  program  work  was  a 
leader  that  was  involved,  that  was  willing 
to  make  a  decision  and  move  forward.  We 
had  good  examples  with  the  first  DJMP 
and  the  AE,  up  front  making  a  decision 
and  going  ahead  and  doing  something 
rather  than  sitting  around  trying  to  figure 
out  the  best  way  to  do  things.  They  fought 
it  out,  made  a  decision,  and  moved  on. 
That  was  critical,  to  have  a  decision  maker 
who  made  a  decision  and  pushed  hard.” 


Conclusions 


Growth  of  Related  Cooperative  Ventures 

The  evidence  indicates,  in  general,  that 
the  two-country  team  has  fostered  strong 
relationships  based  on  technical  expertise 
and  trust,  plus  both  countries  had  some¬ 
thing  to  offer  the  other.  The  payoff  to  both 
has  been  additional  cooperative  efforts. 

To  date  the  U.S.  JPO  has  negotiated  and 
signed  an  MOU  amendment  to  further  co¬ 
operate  with  the  U.K.  MoD  on  an  ad¬ 
vanced  Missile  Warning  System  (MWS) 
technology  assessment  program.  Work 
under  this  MOU  amendment  may  yield 
technology  that  could  be  used  in  the 
planned  upgrade  program  for  the  baseline 
DIRCM  missile  warning  sensor. 

Further,  the  U.S.  JPO  is  in  the  final 
stages  of  nego- 

"The  evidence 
indicates,  in  general, 
ment  to  cooper-  that  the  two«country 
ate  on  a  laser  team  has  festered 
technology  as-  strong  relationships 


dating  another 
MOU  amend- 


based  on  technical 
expertise  and  trust, 
plus  both  countries 
had  something  to 
offer  the  other." 


sessment  pro¬ 
gram.  As  with 
the  advanced 
MWS,  technol¬ 
ogy  developed 
here  could  be 

transitioned  into  the  planned  upgrade  to 
add  a  laser-based  countermeasure. 


Lessons  Learned 

The  DIRCM  program  was  researched 
so  that  future  international  cooperative  ac¬ 
quisitions  may  directly  benefit  from  it’s 
experiences.  Each  of  the  29  people  inter¬ 
viewed  for  the  study  offered  lessons  per¬ 
tinent  to  their  areas  of  interest,  the  more 
important  of  which  are  summarized  above. 


369 


Acquisition  Review  Quarterly — Fall  1998 


The  lessons  that  the  majority  of 
interviewees  agree  on  collectively  are 
synopsized  below  to  aid  the  reader  look¬ 
ing  for  this  article’s  “bottom  line”  of 
successful  international  collaboration. 
Programmatic  conclusions  follow. 

•  Each  side  must  strongly  desire  to 
achieve  the  same  well-defined,  focused 
requirement  while  being  able  to  offer 
some  technical  benefit  to  the  other  side. 
In  general,  if  one  or  more  of  the  MOU 
participants  appear  to  have  little  or  no 
technological  benefit  to  offer  the  re¬ 
maining  participant(s),  the  mismatch, 
real  or  perceived,  will  not  allow  the 
cooperative  effort  to  get  off  the  ground. 

•  Each  side  must  dedicate  an  enormous 
effort  to  understand  the  culture,  moti¬ 
vations,  and  idiosyncrasies  of  the 
people  and  bureaucracies  of  the  other 
country. 

•  Strong  personalities  and  technical  com¬ 
petence  are  essential  in  the  initial 
contact  and  management  teams. 

•  International  cooperative  programs  can 
achieve  increased  synergistic  results  if 
the  managers  take  care  to  staff  their 
technical  teams  to  complement,  rather 
than  mirror,  each  other. 

•  When  a  junior  partner,  the  U.S .  side  must 
be  prepared  to  accept  a  subordinate  man¬ 
agement  role.  We  note  an  overall  theme 
that  emerged  from  D’Agostino’s  (1996) 
research  “for  success  in  multinational 
programs  that  have  been  well-selected, 
national  political  issues  and  pride  need 
to  be  subordinated  to  what  is  best  for  the 
program.” 


•  Precoordinate  contract  technical  and 
testing  terms,  conditions,  and  standards 
among  all  involved  agencies. 

•  U.S.  DJPMs  should  plan,  as  much  as 
possible,  on  conducting  their  side  of  the 
program  consistent  with  the  DoD  5000 
Series,  even  though  a  Section  27 
“Quayle”  Authority  program  may  be 
conducted  in  accordance  with  the  other 
country’s  laws  and  acquisition  proce¬ 
dures.  This  approach  will  pay  off  in  the 
short  and  long  term  by  providing  the 
required  information  in  a  familiar 
format  to  the  appropriate  staff  agencies. 

•  The  U.S.  acquisition  system  has  expe¬ 
rienced  a  revolution  through  recent  ac¬ 
quisition  reform  initiatives.  However, 
U.S.  personnel  should  avoid  the  ten¬ 
dency  to  see  their  acquisition  system 
as  superior  to  all  others.  It  is  working 
under  a  unique  set  of  circumstances, 
statutes,  industry  capabilities,  and 
congressional  oversight. 

•  Make  effective  use  of  priced  produc¬ 
tion  and  sustainment  options  during  the 
Competition.  The  United  Kingdom 
awarded  development  and  production 
together,  giving  up  significant  leverage 
in  the  process. 

•  Look  closely  at  manning  decisions 
when  creating  program  offices.  Each 
country’s  program  office  should  have 
a  representative  from  the  other  coun¬ 
try.  Consideration  should  be  given  to 
prime  contractors  and  major  test  fa¬ 
cility  locations  when  making  final 
manning  decisions. 
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Programmatic  Conclusions 

While  the  DIRCM  program  has  en¬ 
joyed  noteworthy  success  in  its  Section 
27  “Quayle”  Authority  environment  to 
date,  the  interviews  hint  that  not  all  has 
been  smooth  and  trouble-free — the  pro¬ 
gram  experiences  its  share  of  technical 
problems.  The  evidence  suggests  that 
DIRCM  was  acknowledged  and  formu¬ 
lated  as  a  high-performance  and  schedule- 
risk  program.  Urgent  warfighter  need  for 
infrared  missile  protection  by  USSOCOM 
and  U.K.  MoD  aviation  units  dictated  a 
truncated,  perhaps  ambitious  acquisition 
schedule.  That  schedule  appears  to  am¬ 
plify  technical  problems  which  may  oth¬ 
erwise  be  classified  as  typical  of  an 
acquisition  program  at  this  stage. 

Untypical,  however,  is  the  teamwork 
approach  of  the  program’s  people  in  an¬ 
ticipating  emerging  problems  and  wres¬ 
tling  them  through  to  resolution.  Team¬ 
work  and  a  program-office-wide  work 
ethic — the  desire  to  succeed  and  achieve — 
led  us  to  examine  perhaps  the  central  rea¬ 
son  for  the  program’s  success:  organiza¬ 
tional  character.  We  thought  future  inter¬ 
national  cooperative  program  managers 
might  be  interested  in  this  notion  when 
establishing  their  program  offices. 

In  our  view,  the  program’s  originators 
established  a  core  ideology  (values  and 
purpose)  that  has  been  foundational  to 
DIRCM’s  success  in  its  unique,  relatively 
complex.  Section  27  environment.  Ac¬ 
cording  to  Collins  and  Porras  (1996),  core 
ideology — defined  as  the  enduring  char¬ 
acter  of  an  organization — is  the  most  last¬ 
ing  and  significant  contribution  of  those 
who  build  visionary  organizations.  Core 
ideology  provides  the  glue,  the  consistent 
identity,  that  holds  an  organization  to¬ 
gether  through  time.  Core  ideology  is  not 


intentionally  created  or  set;  one  discovers 
core  ideology.  In  their  model,  core  ideol¬ 
ogy  has  two  distinct  parts:  core  values,  a 
system  of  guiding  principles  and  tenets; 
and  core  purpose,  the  organization’s  most 
fundamental  reason  for  existence. 

In  the  case  of  DIRCM,  core  value  is  the 
program  organization’s  spirit  or  culture  of 
teamwork.  The  teamwork  culture  was 
started  by  the  OSD,  U.K.  MoD,  AFSOC, 
and  USSOCOM  founders — who  were 
determined  to  work  and  succeed  together 
as  a  team — and  continues  in  the  program 
today.  With  few 

exceptions,  "Cere  ideology 
members  of  the  provides  the  glue, 
DIRCM  team  the  consistent 
have  put  team-  identity,  that  holds 
work  above  any  **"  organization 
company,  Ser-  *hrov*h 

vice,  promotion, 
or  other  paro¬ 
chial  interest.  The  evidence  suggests  that 
without  the  teamwork  spirit  at  its  core,  the 
organization  would  not  have  successfully 
expanded,  through  IPT  management,  to 
involve  the  many  geographically  and 
functionally  scattered  stakeholders. 

Core  purpose ,  DIRCM’s  most  funda¬ 
mental  reason  for  existence,  is  urgent 
warfighter  protection.  Purpose,  not  to  be 
confused  with  programmatic  goals  or 
strategies,  is  the  vision  of  a  light  beam 
defeating  an  enemy  missile  seconds  out 
from  destroying  an  aircraft  and  its  crew. 
With  its  roots  mainly  in  the  memory  of  a 
Special  Operations  aircraft  downed  in  the 
Gulf  War,  purpose  has  held  the 
organization’s  key  members  together 
through  several  disruptive  and  divisive 
business  and  government  reorganizations. 
There  are  no  indicators  it  will  abate  prior 
to  fielding  and  sustainment. 
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LESSONS  LEARNED 


THE  COSTS  AND  BENEFITS 
OF  THE  EARNED  VALUE 
MANAGEMENT  PROCESS 

DavidS.  Christensen,  Ph.D. 


Up  to  now  assessments  of  the  earned  value  management  process  have 
focused  on  either  the  positives  or  the  negatives— but  a  complete  and  objective 
evaluation  has  not  been  made.  This  review  surveys  the  literature  and  paints  a 
balanced  picture  of  the  issue. 


Do  the  benefits  of  the  earned  value 
management  process  exceed  its 
costs?  Several  published  studies 
report  the  costs  but  ignore  the  benefits. 
Others  focus  on  the  benefits,  but  ignore 
the  costs. 

A  widely  circulated  report  by  Coopers 
&  Ly brand  and  TASC  (1994),  for  ex¬ 
ample,  concludes  that  the  Department  of 
Defense  regulatory  cost  is  significant,  and 
the  requirement  for  earned  value  manage¬ 
ment  systems  (EVMS)  criteria  is  among 
the  largest  cost  drivers.  But  in  this  study 
the  benefits  of  the  earned  value  informa¬ 
tion  derived  from  criteria-compliant  con¬ 
tractors  were  not  considered.  Clearly,  a 
report  that  addresses  only  the  costs  or  the 
benefits  of  earned  value  can  be  mislead¬ 
ing.  Both  the  costs  and  the  benefits  of 
earned  value  must  be  assessed. 


Here  I  provide  a  comprehensive  litera¬ 
ture  review,  and  summarize  and  synthe¬ 
size  studies  reporting  the  costs  or  the  ben¬ 
efits  of  earned  value.  The  result  is  a  more 
complete  and  objective  evaluation  of  the 
earned  value  management  process. 

Earned  Value  and  EVMS 


There  is  a  difference  between  earned 
value  and  EVMS  criteria.  Earned  value  is 
a  special  metric  that  can  be  used  to  man¬ 
age  any  project.  The  criteria  are  standards 
for  management  control  systems  that  use 
earned  value.  Since  1967  the  criteria  have 
been  required  on  large,  flexibly  priced 
defense  contracts.  The  purpose  of  the  cri¬ 
teria  was  to  assure  the  reliability  of  the 
earned  value  metric.  Although  earned 
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value  does  not  require  the  criteria,  it  does 
require  a  management  control  system  that 
meets  at  least  some  of  the  standards  de¬ 
scribed  by  the  criteria.  In  this  paper,  the 
term  “earned  value  management  process” 
includes  both  earned  value  and  the  EVMS 
criteria. 


Earned  Value 


The  earned  value  concept  originated  in 
industry  and  was  developed  primarily  by 
the  Department  of  Defense  (DoD)  as  a 
management  tool  for  use  on  defense  ac¬ 
quisition  contracts.  Earned  value  is  a  met¬ 
ric  devised  to  achieve  meaningful  com¬ 
parisons  between  planned  and  completed 
work.  It  is  similar  to  what  accountants  call 
a  “flexible  budget,”  where  the  original 
budget  for  work  is  adjusted  for  the  actual 
level  of  output.  Cost  variances  result  when 
the  actual  cost  of  the  work  and  its  flexible 
budget  (earned  value)  differ.  Significant 
variances  are  analyzed  to  identify  and 
correct  problems  before  they  worsen. 

A  major  difference  between  a  flexible 
budget  and  earned  value  is  the  time  di¬ 
mension  associated  with  earned  value. 
Initially,  the  work  on  a  project  is  divided 
into  pieces,  assigned  a  budget,  and  as¬ 
signed  a  schedule.  Because  each  incre¬ 
ment  of  work  is  time-phased,  a  schedule 


variance  occurs  if  work  is  not  completed 
(earned)  when  it  was  scheduled  to  be  com¬ 
pleted.  Because  the  work  has  a  budget,  the 
schedule  variance  is  often  reported  as  a 
dollar  amount.  The  flexible  budget  used 
in  cost  accounting  does  not  provide  any 
information  about  schedule  variances. 
Like  the  cost  variances,  significant  sched¬ 
ule  variances  are  analyzed  and  corrected 
when  possible. 

When  variance  analysis  is  conducted 
properly  (e.g.,  on  time,  and  at  the  proper 
level),  it  can  be  an  effective  control  against 
further  cost  and  schedule  problems  that 
may  jeopardize  the  successfiil  completion 
of  a  project.  Unfortunately,  variance 
analysis  can  be  untimely  or  excessive 
and  even  contribute  to  project  failure  by 
drawing  project  managers,  engineers,  and 
others  away  from  more  urgent  problems. 

EVMS  Criteria 


A  key  to  the  effective  use  of  earned 
value  is  an  adequate  management  control 
system  that  fosters  the  proper  planning  and 
integration  of  work  on  a  project.  EVMS 
criteria  define  the  attributes  that  manage¬ 
ment  control  systems  must  possess  for 
earned  value  to  be  used  effectively.  Origi¬ 
nally,  the  criteria  were  established  by  the 
Air  Force  as  cost/schedule  planning  and 
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control  specifications  (C/SPEC)  for  appli¬ 
cation  on  major  defense  acquisition  con¬ 
tracts.  Later  the  criteria  were  adopted  by 
all  the  military  services  as  cost/schedule 
control  systems  criteria  (C/SCSC).  Re¬ 
cently,  the  criteria  have  been  slightly 
revised  and  renamed  earned  value  man¬ 
agement  control  systems  criteria  (DoD, 
1996). 

Despite  the  multiple  names,  the  crite¬ 
ria  have  not  changed  significantly  since 
their  inception.  Presently,  there  are  32 
EVMS  criteria,  organized  into  five  catego¬ 
ries,  that  pertain  to  major  project  manage¬ 
ment  activities: 

•  organization; 

•  planning  and  budgeting; 

•  accounting; 

•  analysis;  and 

•  revisions. 

Each  criterion  addresses  a  major  prin¬ 
ciple  necessary  for  effective  management 
of  large,  flexibly  priced  defense  projects. 
For  example,  one  criterion  requires  that 
each  element  of  work  on  the  project  has  a 
budget.  Another  criterion  requires  that 
each  element  of  work  has  a  schedule. 
Without  a  budget  and  a  schedule,  it  would 
be  difficult  to  properly  manage  a  project 
of  any  size,  much  less  a  major  defense 
project  that  can  cost  over  a  billion  dollars 
and  last  for  many  years.  Thus,  criteria  are 
often  described  as  common-sense  man¬ 
agement  practices  that  any  well-managed 
defense  contractor  would  use. 

Over  the  years,  however,  implement¬ 
ing  the  criteria  became  an  administrative 


burden  that  was  eventually  viewed  as  a 
non-value-added  activity  by  contractors 
and  program  managers  (Government  Ac¬ 
counting  Office,  1997).  Like  many  gov¬ 
ernment  documents,  the  DoD’s  Joint 
Implementation  Guide  (JIG),  which  de¬ 
scribed  how  to  implement  the  criteria, 
grew  in  size  and  complexity  (DoD,  1987). 
Its  checklist  of  158  questions  (Appendix 
E  of  the  JIG)  was  often  perceived  as  a  con¬ 
tractual  requirement  and  administered 
with  audit-like  rigor  by  the  review  teams. 
Additionally,  earned  value  data  was  mis¬ 
takenly  judged 
“guilty  by  asso- 

ciation”  and  oc-  »E«««h  criterion 

casionally  ig-  addresses  a  major 

nored  by  project  principle  necessary 

managers  who  for  effective  man- 

may  have  ben-  agement  of  large, 

efited  from  it.  #lexlb,Y  Pri«®‘* 

According  to  defense  project*. " 

Abba  (1995),  for 

example,  large  cost  overruns  on  some 
major  defense  projects  were  foreseeable 
from  contractor  earned  value  reports  but 
not  recognized  by  program  managers. 

There  are  several  factors  that  contrib¬ 
uted  to  the  implementation  problem  (e.g., 
a  lack  of  industry  ownership,  inadequate 
training,  and  an  awkward  technical  jar¬ 
gon).  A  major  factor  was  a  failure  in  the 
early  years  to  make  the  earned  value  pro¬ 
cess  the  responsibility  of  program  man¬ 
agers  and  contractors.  B  ased  on  a  two-year 
review  of  the  DoD’s  earned  value  man¬ 
agement  process,  the  GAO  (1997)  con¬ 
cluded  that  while  the  process  was  in¬ 
tended  to  serve  the  needs  of  several  user 
groups,  financial  personnel  managed  the 
process.  It  was  natural  for  this  group  to 
focus  on  their  oversight  responsibilities 
and  stress  criteria  compliance.  But  it  was 
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also  natural  for  other  user  groups,  includ¬ 
ing  the  program  managers,  to  perceive 
earned  value  as  a  purely  financial  report¬ 
ing  requirement.  According  to  the  DoD, 
“the  needs  of  the  program  manager  were 
often  not  met  when  EVMS  were  viewed 
primarily  as  a  financial  reporting  system” 
(GAO,  1997,  p.  29). 

Through  the  years,  the  criteria  imple¬ 
mentation  problem  has  prompted  studies 
that  addressed  either  the  benefits  or  the 
costs  of  the  earned  value  management  pro¬ 
cess.  In  general,  studies  that  focused  on 
benefits  concluded  that  earned  value  and 
the  criteria  concept  were  sound,  while 

those  focused 
on  cost  reported 
the  cost  of  com¬ 
pliance  to  be 
relatively  small, 
ranging  from 
less  than  one  to 
five  percent  of 
contract  cost.  I 
found  no  study 
that  directly  compared  benefits  with  costs, 
possibly  because  the  benefits  are  largely 
nonquantifiable.  Clearly,  any  study  that 
focuses  on  only  one  side  of  a  cost-benefit 
issue  may  be  misleading. 

Regardless  of  the  focus,  nearly  all  of 
the  studies  are  unpublished,  thus  contrib¬ 
uting  to  the  difficulty  of  comparing  costs 
with  benefits.  My  purpose  is  to  remedy 
this  problem  by  providing  a  summary  of 
all  I  have  found.  I  have  compiled  and  re¬ 
viewed  most  of  the  cost  and  benefit  stud¬ 
ies  related  to  earned  value.  The  results  of 
this  survey  follow,  as  does  a  conceptual 
framework  for  comparing  the  costs  and 
benefits  of  the  EVM  process. 


The  Costs  of  the  IVM  Process 

Most  of  the  cost  studies  reviewed  ap¬ 
propriately  focus  on  the  incremental  cost 
of  EVMS  compliance  and  reporting.  The 
“normal”  costs  of  operating  a  management 
control  system  are  not  considered  relevant 
because  they  would  be  incurred  in  the 
absence  of  any  requirement  for  an  earned 
value  management  system  (DoD,  1987, 
p.  viii). 

Coopers  &  Lybrand/TASC 

The  most  recent  study  of  this  kind,  “The 
DoD  Regulatory  Cost  Premium:  A  Quan¬ 
titative  Assessment,”  was  conducted 
jointly  by  Coopers  &  Lybrand  and  TASC 
(C&L/TASC)  under  the  auspices  of  the 
Under  Secretary  of  Defense  for  Acquisi¬ 
tion  and  Technology  (1994).  Its  purpose 
was  to  estimate  the  industry  cost  of  DoD 
regulation  and  oversight,  including  the 
regulatory  requirement  for  EVMS.  It  did 
not  include  the  DoD’s  direct  oversight 
costs  (e.g.,  government  auditors).  Based 
on  an  analysis  of  10  contractor  facilities, 
activity  based  costing  was  used  to  report 
an  average  regulatory  cost  premium  of  18 
percent  of  value-added  costs.1  The  cost  of 
EVMS  ranked  third  among  the  top  10  cost 
drivers,  and  was  estimated  to  be  about  0.9 
percent  of  the  value-added  costs.  For  ex¬ 
ample,  in  a  graph  depicting  value-added 
costs,  C&L/TASC  indicate  that  material 
purchases  are  about  40  percent  of  the  cost 
of  a  contract  (p.  4a).2  Thus,  on  a  $100 
million  contract,  $60  million  (60  percent) 
would  be  value-added,  $10.8  million 
(18  percent  of  $60  million)  would  be  the 
regulatory  premium  and  $0.54  million 
(0.9  percent  of  $60  million)  would  be  the 
regulatory  cost  of  EVMS  to  industry. 


"I  found  no  study 
that  directly  com-  H 
pared  benefits  with 
costs,  possibly  be¬ 
cause  the  benefits 
are  largely 
nonquantifiable/' 
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Most  of  the  EVMS  costs  were  in  the 
areas  of  engineering  and  program  man¬ 
agement  (65  percent),  and  finance  (25 
percent).  Program  managers  are  respon¬ 
sible  for  the  entire  management  control 
system.  Engineers  are  typically  respon¬ 
sible  for  variance  analysis  and  reporting. 
Finance  personnel  are  typically  required 
to  ensure  compliance  with  the  EVMS  cri¬ 
teria.  A  DoD  working  group  that  investi¬ 
gated  these  costs  reported  that  most  of 
them  (two-thirds  to  three-fourths)  were 
unnecessary  and  not  required  by  criteria 
(DoD,  1997,  pp.  8-9).  An  example  is  pre¬ 
paring  written  variance  reports  at  detailed 
levels  in  the  work  breakdown  structure, 
termed  “control  accounts.”  Although  the 
criteria  do  not  require  a  written  variance 
analysis  for  each  control  account,  govern¬ 
ment  review  teams  came  to  expect  them 
as  tangible  evidence  of  criteria  compli¬ 
ance.  Contractors  with  written  variance 
reports  were  more  likely  to  be  found  com¬ 
pliant  than  contractors  without  written 
variance  reports.  Eventually,  a  written 
variance  report  for  every  control  account 
became  an  unwritten  rule.  Other  examples 
of  non- value-added  activities  that  came  be 
to  expected  for  criteria  compliance  are 
provide  by  Abba  (1997,  p.  3). 

Clearly,  the  0.9  percent  EVMS  cost  pre¬ 
mium  is  only  a  rough  estimate.  The  cost 
premium  would  be  greater  if  the  direct 
government  costs  were  included.  The  pre¬ 
mium  would  be  smaller  if  the  activities 
not  required  by  the  criteria  were  elimi¬ 
nated.  In  addition,  C&L/TASC  warn 
against  generalizing  the  results  of  their 
study  (p.  3).  The  sample  was  nonrandom 
and  only  2  of  the  10  sites  had  contracts 
subject  to  EVMS  criteria.  Further,  the 
results  cannot  be  independently  verified 
because  the  data  are  proprietary.  In 


general,  C&L/TASC  concluded  that  the 
contractors  viewed  EVMS  positively,  but 
that  it  was  costly  to  implement  (p.  22): 


All  contractors  subject  to  C/SCSC 
(EVMS  criteria)  agree  that,  as 
currently  required  by  DoD,  cost- 
schedule  reporting  is  too  detailed, 
repetitive,  and  voluminous  to  be 
used  effectively  as  a  management 
tool  by  either  the  government  or 
industry,  and  that  the  requirement 
may  in  fact  undermine  program 
performance  by  diverting  the 
time  and  attention  of  the  company 
program  manager. 


Other  Cost  Studies 


Lampkin  (1992)  reviewed  five  studies 
that  estimate  the  marginal  cost  of  imple¬ 
menting  and  maintaining  a  criteria-com- 
plaint  EVMS  (Table  l).3  The  cost  range  is 
expressed  as  a  percentage  of  contract  cost. 
The  first  three  estimates  are  based  on  opin¬ 
ion  surveys  of  industry  pr  DoD  experts. 


"Eventually,  a 
written  variance 
report  for  every 
control  account 
became  an 
unwritten  rule." 


The  MITRE  es¬ 
timate  pertains 
to  Air  Force 
software  devel¬ 
opment  contract 
only.  Decision 
Planning  Cor- 
poration  (1992) 
uses  a  cost-esti-  . 

mating  model  that  assumes  a  generic,  3- 
year  research  and  development  contract 
of  $75M.  Humphreys  and  Associates  dis¬ 
tinguish  between  nonrecurring  and  recur¬ 
ring  costs.  Nonrecurring  costs  pertain  to 
implementing  a  criteria-compliant  system 
for  the  first  time,  and  range  from  2.5  to 
4.0  percent.  Recurring  costs  pertain  to 
maintaining  criteria-compiiance,  and 
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Table  1. 

The  Marginal  Cost  of  EVMS  Criteria 
(Percent  of  Contrast  Cost) 


Author  (Year) 

Source  of  Estimate 

Cost  Range  (%) 

Kouts  (1 978) 

Survey  of  industry 

0.5  to  5 

MITRE  Corp.  (1982) 

Survey  of  industry 

0.1  to  0.2 

DoD  IG  (1984) 

Survey  of  DoD  experts 

5.0 

Decision  Planning  Corp.  (1992) 

Industry  cost  estimation  model 

0.6  to  1.0 

Humphreys  and  Associates  (1992) 

Consultant  experience 

0.5  to  4.0 

Lampkin  (1992) 

Average  of  five  studies  above 

0.4  to  1.63 

Lampkin  (1992) 

range  from  0.5  to  1 .5  percent.  The  final  of  its  purposes  was  to  determine  where 
row  in  the  table  is  an  average  of  these  there  may  be  excessive  cost  and  to  what 
estimates,  also  reported  by  Lampkin  (p.  31). 4  extent  the  cost  could  be  reduced .  The  team 

concluded  that  there  are  excesses  in 
UNNECESSARY  Cost  Drivers  EVMS  implementation  and  reporting  that 

Survey  and  interview  research  indicate  result  in  unnecessary  cost  (p.  3.9.7). 
that  at  least  some  of  the  cost  of  EVMS  is 

unnecessary  and  due  to  an  over-implemen-  EVMS  REFORMS  AND  COST  REDUCTION 
tation  of  the  criteria  (Table  2).  Examples  The  sensational  cancellation  of  the 
include  lengthy  “system  descriptions”  of  Navy’s  A-12  program  (Beach,  1990)  and 
EVMS,  written  variance  analysis  at  the  a  subsequent  DoD  audit  report  (1993) 

control  account  level,  and  over-specified  were  additional  catalysts  for  reform.  To 

work  breakdown  structures.  The  National  promote  a  program  management  orienta- 

Security  Industrial  Association  (NSIA)  tion,  the  DoD  policy-making  body  for 

(1980,  p.  17)  estimated  the  number  of  EVMS  was  shifted  to  an  executive  steer- 

pages  required  to  achieve  and  maintain  ing  group  with  representatives  from  the 

criteria-compliance  for  the  industry  to  be  services,  acquisition  executives,  and  the 

32.8  million  pages  annually.  Based  on  a  Defense  Contract  Management  Command 

survey  and  interviews,  Arthur  D.  Little  (DCMC)  in  1995.5  To  refocus  program 

(1983,  1984)  concluded  that  EVMS  was  management  attention  to  the  informa- 

a  good  approach  to  controlling  contract  tion  in  earned  value  reports,  compliance 

performance,  but  that  there  was  room  for  responsibility  was  transferred  from  the 

improvement  (p.  1-3).  military  services  to  DCMC  in  1996.  Fi- 

Prompted  by  this  conclusion,  the  DoD  nally,  the  DoD  criteria  used  since  1967 

and  NSIA  formed  a  total  quality  manage-  were  replaced  with  industry  standards 

ment  team  to  review  EVMS  in  1989.  One  in  1 996,  and  the  possibility  for  industry 
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Table  2. 

Other  Studies  Related  to  the  Cost  of  EVMS 


Author  (Year) 

Research  Method  (Sample  Size) 

Cost  Drivers  Related 
to  Over- Implementation 

NSIA  (1980) 

Opinion  survey  (74  contractors 

Excessive  documentation 

A.D.  Little  (1984) 

Interview  (56  managers) 

Excessive  levels  of  detail  in  the  WBS 

DoD/NSIA  (1991) 

Interview  (250  mangers) 

Written  variance  analysis  reports 

self-certification  was  offered  (Christie, 
1994). 

Removing  unnecessary  requirements 
related  to  EVMS  will  likely  reduce  but  not 
eliminate  the  marginal  cost.  The  NSIA  es¬ 
timated  industry-wide  savings  from  re¬ 
forming  EVMS  could  reach  over  one  bil¬ 
lion  dollars  annually  (Christie,  1996).  A 
DoD  working  group  estimates  the  EVMS 
regulatory  premium  can  be  reduced  by  one 
third  or  0.3  percent  of  value-added  costs 
(DoD,  1997).  Thus,  two-thirds  of  the 
EVMS  regulatory  premium  or  0.6  percent 
will  remain. 


The  Benefits  of  the  IVM  Process 

None  of  the  marginal  cost  studies  de¬ 
scribes  the  marginal  benefits  of  EVMS, 
perhaps  because  the  benefits  are  difficult 
to  quantify.  Accordingly,  most  benefit 
studies  I  review  here  are  qualitative 
assessments.  However,  to  the  extent  that 
the  criteria  help  a  company  use  or  con¬ 
tinue  to  use  the  management  principles 
required  by  the  criteria,  I  believe  the 
marginal  benefits  of  the  criteria  are  greater 
than  zero. 


The  Legacy  of  EVMS 

The  full  application  of  the  criteria  is  ap¬ 
propriate  for  large,  cost-reimbursable  con¬ 
tracts  where  the  government  bears  the  cost 
risk.  For  such  contracts,  the  management 
discipline  described  by  the  criteria  is  es¬ 
sential.  The  box  on  following  page  is  an 
abbreviated  list  of  EVMS  benefits,  de¬ 
scribed  by  Fleming  and  Koppelman  as  the 
legacy  of  using  the  criteria  on  government 
contracts  for  three  decades  (1996,  p.22). 
Note  that  they  do  not  separate  the  benefits 
of  earned  value  data  from  the  benefits  of 
the  criteria,  perhaps  because  the  reliabil¬ 
ity  of  data  depends  on  the  disciplined  ap¬ 
plication  of  the  management  practices  de¬ 
scribed  by  the  criteria. 

Benefit  1.  Although  the  criteria  do  not 
require  an  external  report,  managing  with 
one  system  while  reporting  from  another 
is  neither  efficient  nor  effective.  The  cri¬ 
teria  concept  encourages  the  company  to 
use  its  own  internal  management  control 
systems,  provided  those  systems  meet  the 
management  standards  described  by  the 
criteria. 

Benefit  2.  The  criteria  require  that  all 
the  authorized  work  and  related  resources 
are  defined  and  integrated  using  a  prod¬ 
uct-oriented  work  breakdown  structure. 
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Ten  Benefits  of  EVMS 

1 .  It  is  a  single  management  control  system  that  provides  reliable  data. 

2.  It  integrates  work,  schedule,  and  cost  using  a  work  breakdown  structure. 

3  The  associated  database  of  completed  projects  is  useful  for  comparative  analysis. 

4.  The  cumulative  cost  performance  index  (CPI)  provides  an  early  warning  signal. 

5.  The  schedule  performance  index  provides  an  early  warning  signal. 

6.  The  CPI  is  a  predictor  for  the  final  cost  of  the  project. 

7.  It  uses  an  index-based  method  to  forecast  the  final  cost  of  the  project. 

8.  The  “to-complete”  performance  index  allows  evaluation  of  the  forecasted  final  cost. 

9.  The  periodic  (e.g.,  weekly  or  monthly)  CPI  is  a  benchmark. 

1 0.  The  management  by  exception  principle  can  reduce  information  overload. 


For  a  company  that  has  managed  by  func¬ 
tional  areas  only  (e.g.,  engineering,  manu¬ 
facturing,  accounting),  the  product  orien¬ 
tation  can  help  organize  and  coordinate  the 
contributions  of  each  area,  and  ensure  that 
work,  schedule,  and  cost  are  properly 
integrated. 

Benefit  3.  The  consistent  reporting  of 
criteria-compliant  projects  for  over  30 
years  has  resulted  in  a  database  useful  for 
comparative  analysis  analysts  have  used 
this  database  to  create  important  insights 
for  managers.  For  example,  a  compara¬ 
tive  analysis  of  the  cost  performance  of 
similar  aircraft  was  compelling  evidence 
that  the  Navy’s  A-12  project  was  out  of 
control  (Beach,  1990). 

Benefit  4.  The  cumulative  cost  perfor¬ 
mance  index  (CPI),  defined  as  the  earned 
value  to-date  divided  by  the  cost  to-date, 
has  been  shown  to  stabilize  to  within  10 
percent  by  the  20  percent  completion  point 


for  most  defense  acquisition  contracts.  In 
most  cases  the  cumulative  CPI  only 
worsens  (Christensen  and  Heise,  1993). 
Among  other  things,  this  indicates  that 
cost  management  must  occur  early  to  be 
effective. 

Benefit  5.  The  schedule  performance 
index  (SPI),  defined  as  earned  value 
divided  by  planned  value,  is  useful  for 
identifying  schedule  problems,  especially 
when  used  with  critical  path  information 
(Fleming  and  Koppelman,  1996,  p.  5).  Be¬ 
cause  schedule  problems  are  often  re¬ 
solved  by  additional  spending,  an  adverse 
SPI  is  also  predictive  of  later  cost  prob¬ 
lems.  The  criteria  recommend  that  all  the 
work  is  scheduled  and  traceable  from  the 
master  program  level  to  the  detailed  lev¬ 
els.  Consistent  with  the  criteria  concept, 
no  specific  scheduling  system  is  required. 

Benefit  6.  The  cumulative  CPI  is  also 
useful  for  determining  a  reasonable  lower 
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limit  for  the  estimated  final  cost  of  a  con- 
tract,  termed  the  estimate  at  completion 
(EAC)  (Christensen,  1996).  A  lower 
bound  is  useful  for  planning  and  control 
purposes.  The  criteria  recommend  that  the 
estimate  be  evaluated  regularly. 

Benefit  7.  It  has  been  shown  that  the 
SPI  and  CPI  can  be  combined  to  estimate 
a  reliable  upper  bound  to  the  EAC 
(Christensen,  1996).  When  combined  with 
the  CPI-based  lower  bound,  a  “most 
likely”  range  of  EACs  is  determined. 
When  the  contractor’s  EAC  is  outside  this 
range,  there  may  be  a  problem  with  the 
contractor’s  estimation  system. 

Benefit  8.  Another  earned  value  index, 
the  to-complete  performance  index 
(TCPI),  is  useful  for  evaluating  the  rea¬ 
sonableness  of  the  contractor’s  EAC  or 
other  financial  goals  (Christensen,  1994). 
The  TCPI  is  the  ratio  of  the  remaining 
work  to  the  remaining  financial  resources. 
It  indicates  the  level  of  performance  that 
the  contractor  must  achieve  to  reach  a 
financial  goal.  Thus,  this  earned  value 
metric  can  help  the  manager  assess  the  rea- 
sonableness  of  critical  financial  goals, 
such  as  completing  the  remaining  work 
within  the  targeted  cost. 

Benefit  9.  While  cumulative  perfor¬ 
mance  indices  are  useful  for  predicting 
trends  at  summary  levels  in  the  work 
breakdown  structure  (WBS),  weekly  or 
monthly  CPIs  are  useful  for  cost  perfor¬ 
mance  trends  at  the  detailed  levels  of  the 
WBS  (Fleming  and  Koppelman,  p.  28). 
The  criteria  recommend  an  analysis  of 
these  and  all  other  metrics  at  the  frequency 
and  level  needed  by  management  for 
effective  control  (DoD,  1996). 

Benefit  10.  By  directing  management 
attention  to  only  the  most  critical  problems, 
information  overload  can  be  reduced. 


Although  not  always  implemented  prop¬ 
erly,  the  criteria  encourage  variance 
thresholds  and  tailoring  to  reduce  the  po¬ 
tential  for  overload. 

Other  benefit  studies.  Survey  research 
shows  that  most  managers  agree  that 
EVMS  has  benefits  (NSIA,  1980;  Little, 
1983  and  1984;  DoD/NSIA,  1991).  Based 
on  a  survey  of  534  managers,  for  example, 
more  than  70  percent  agree  that  “a  major 
benefit  of  the 
criteria  is  more 
thorough  plan¬ 
ning  than  would 
otherwise  be  ac- 
complished,” 
and  that  EVMS 
is  “effective  in  ? 

helping  managers  control  contract  perfor¬ 
mance”  (Little,  1983,  0.  El-3).  Govern¬ 
ment  audits  of  the  EVMS  process  have  not 
challenged  this  perception  (DoD,  1993; 
GAO,  1997).  For  example,  the  GAO 
(1997)  reports  that  the  earned  value  con¬ 
cept  is  “recognized  as  a  sound  way  to 
measure  progress  on  major  acquisition 
programs”  (p.  3). 


"Survey  research 
shows  that  inoif 
managers  agree 
that  JEVM^  has 
benefits." 


Synthesis 


Figure  1  identifies  qualitative  charac¬ 
teristics  that  a  report  should  possess  to  be 
useful  for  decision  making.6 1  believe  it 
is  a  useful  conceptual  framework  for 
comparing  the  costs  and  benefits  of 
EVMS.  The  output  of  the  EVMS  process 
is  the  earned  value  report.  To  be  useful 
for  decision  making,  the  report  should 
have  relevance  and  reliability. 

To  be  relevant,  the  report  should  have 
predictive  value,  feedback  value,  and  be 
timely.  Many  of  the  marginal  benefits  of 
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incurred  by 
Oversight-community 


Figure  1.  The  Costs  and  Benefits  of  EVMS 


EVMS  pertain  to  these  characteristics.  The 
benefit  studies  indicate  that  managers  and 
oversight  personnel  find  the  EAC  and  the 
variance  information  useful,  if  not  timely. 
Surveys  indicate  some  dissatisfaction  with 
the  report  being  too  late  (DoD/NSIA, 
1991);  however,  recent  improvements  in 
computer  technology  and  software  have 
made  real-time  access  to  contractor  data¬ 
bases  achievable  (GAO,  1997). 

To  be  reliable,  the  report  should  be  veri¬ 
fiable,  valid,  and  objective.  Oversight 
personnel  are  primarily  concerned  with 
reliability.  The  marginal  costs  of  EVMS 
arise  primarily  from  the  need  for  reli¬ 
ability.  The  government  needs  access  to 
contractor  data  verify  its  accuracy.  The 
government  also  needs  assurance  that  the 
contractor  is  reporting  and  managing 
from  the  same  database.  Finally,  because 
the  criteria  are  applied  primarily  to  cost- 


reimbursable  contracts,  the  cost  data  must 
be  objective. 

Achieving  these  qualitative  character¬ 
istics  is  not  easy.  The  double-headed  ar¬ 
row  between  relevance  and  reliability  sug¬ 
gests  a  tradeoff.  A  report  may  not  be 
timely,  for  example,  if  every  number  must 
be  verified.  The  GAO  (1997)  observed 
that  a  delicate  balance  exists  between  man¬ 
agers  (needing  relevance)  and  the  over¬ 
sight  community  (needing  reliability),  and 
concluded  that  because  the  oversight  com¬ 
munity  managed  the  EVMS  process,  the 
need  for  reliability  was  stressed  over  the 
need  for  relevance. 

However,  EVMS  reforms  have  restored 
the  balance.  The  marginal  cost  of  EVMS 
has  decreased,  while  the  marginal  ben¬ 
efits  have  increased.  As  the  unnecessary 
activities  related  to  implementing  the 
criteria  (e.g.,  written  variance  reporting) 
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are  eliminated,  the  DoD  estimated  a  one- 
third  reduction  in  the  EVMS  cost  pre¬ 
mium.  With  the  shift  in  ownership  and 
management  responsibilities  to  program 
managers,  the  value  of  the  earned  value 
reports  should  be  easier  for  program 
managers  to  recognize  (Abba,  1997). 

Ultimately,  the  decision  of  whether  the 
marginal  benefits  of  EVMS  exceed  the 
marginal  cost  is  subjective.  Perhaps  the 
most  compelling  evidence  that  benefits 
exceed  the  costs  is  the  astonishing  increase 


to  earned  value  outside  the  DoD  by  other 
agencies,  commercial  companies,  and 
other  countries  (Abba,  1997;  GAO,  1997). 
After  decades  of  assertions  that  EVMS 
is  too  expensive  and  only  appropriate 
for  DoD  projects,  earned  value  is  rec¬ 
ognized  more  than  ever  as  a  necessary  and 
effective  management  tool  for  projects  of 
any  size  and  risk.  The  key  is  the  proper 
application  of  EVMS  (Fleming  and 
Koppelman,  1996). 
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Endnotes 


1 .  Value-added  costs  are  the  costs  to  con¬ 
vert  raw  material  to  a  finished  prod¬ 
uct.  Total  costs  were  not  used  as  the 
base  to  determine  the  regulatory  pre¬ 
mium  because  a  prime  contractor’s 
material  purchases  are  “to  a  great  ex¬ 
tent  the  value-added  costs  of  its  sub¬ 
contractors  and  suppliers.”  Using  to¬ 
tal  costs  in  the  denominator  would 
double-count  material  costs  and  un¬ 
derstate  the  regulatory  cost  impact 
(Coopers  &  Lybrand  and  TASC,  1994, 
p.  4). 

2.  C&L/TASC  did  not  indicate  that  40 
percent  was  based  on  the  actual  cost 
data  from  the  sample.  It  may  simply 
be  an  example  used  to  explain  the 
meaning  of  value-added  costs. 

3.  Unfortunately,  we  could  not  locate 
most  of  these  studies  to  evaluate  the 
methodology. 


4.  C&L/TASC  used  value-added  costs  to 
determine  the  0.9  percent  cost  pre¬ 
mium  of  E  VMS .  If  the  percentages  in 
Table  1  were  also  based  on  value- 
added  cost,  they  would  be  larger.  For 
example,  if  the  average  value-added 
cost  is  60  percent  of  contract  cost,  the 
range  reported  by  Lampkin  would  be 
0.67  percent  to  2.71  percent. 

5 .  As  the  government’s  on-site  represen¬ 
tative  at  the  contractor’s  facility, 
DCMC  provided  DoD  with  assess¬ 
ments  of  the  contractor’s  perfor¬ 
mance. 

6.  We  have  adapted  this  from  a  concep¬ 
tual  framework  used  by  the  Financial 
Accounting  Standards  Board  (1980). 
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OPINION 


A  CASE  STUDY 
FOR  THE  SYSTEMS 
APPROACH  FOR 
DEVELOPING  CURRICULA 

"DON!  THROW  OUT 
THE  BABY  WITH 
THE  BATH  WATER" 

Dr.  Anthony  A.  Stafati 


Instructional  systems  design  (ISD)  is  a  systematic  model  used  to  plan,  design, 
develop,  and  evaluate  training.  The  Defense  Acquisition  University  consortium 
uses  ISD  to  develop  and  revise  curriculum.  If  we  agree  that  all  learning  manifests 
itself  by  observable  behaviors,  then  we  can  measure  the  progress  and 
effectiveness  of  training.  The  author  urges  that  ISD  is  especially  effective  in 
developing  learning  experiences  that  meet  the  needs  of  acquisition  community. 
It  does  not  hamper  “academic  freedom,”  but  instead  provides  consistency 
and  performance  standards — both  necessary  for  the  move  to  distance  learning 
and  computer-based  instruction. 


Ruth  Colvin  Clark  (1989,  p.  3)  de¬ 
fines  instructional  systems  design 
(ISD)  as  a  “...systematic  model 
used  to  plan,  design,  develop,  and  evalu¬ 
ate  training.”  There  are  many  ISD  models 
in  existence  today,  but  all  have  a  varia¬ 
tion  of  the  following  characteristics.  They 
incorporate: 


a  needs  analysis; 
a  task  analysis; 

a  definition  of  learning  objectives; 
the  development  of  an  assessment  plan; 
the  development  of  learning  material; 
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•  a  plan  to  try  out  with  revision  (pilot); 

and 

•  the  implementation  of  the  final  product 

(Clark,  1989). 

The  systems  approach  models  are  the 
result  of  more  than  25  years  of  research 
in  the  learning  process  (Dick  &  Carey, 
1990).  It  is  used  throughout  industry  and 
government  and  in  academia.  It  is  not  the 
only  way  to  develop  curriculum,  but  it  is 
a  proven  and  effective  model.  The  Defense 
Acquisition  University  (DAU)  and,  con¬ 
sequently,  the  DAU  consortium  schools, 
use  their  version  of  ISD  to  develop  and 
revise  their  curriculum.  The  latest  guid¬ 
ance  concerning  the  DAU  methodology 
is  found  in  the  “Defense  Acquisition 
Guide  for  Curriculum  Development,  De¬ 
livery,  and  Evaluation”  (November,  1997), 
which  can  be  found  on  the  DAU  Home 
Page  (http:Wwww.acq.osd.mil/dau). 

The  systems  approach  to  designing  cur¬ 
riculum  has  many  names:  Instructional 
Systems  Design  (MilStd  1379D)— the  old 
military  standard,  The  Critical  Events 
Model  (Nadler,  1982),  Systematic  Design 
of  Instruction  (Dick  &  Carey,  1990),  the 
DAU  Guide  for  Curriculum  Development, 
Delivery,  and  Evaluations  (November, 
1997),  and  a  host  of  others.  They  all  are 
based  on  a  common  belief  that  all  learning 


manifests  itself  by  observable  behaviors 
in  the  psychomotor,  cognitive,  or  affective 
domains.  And  if  there  is  any  credence  to 
the  metaphor  that  learning  is  a  journey, 
then  with  ISD  we  can  describe  the  end 
state  and  measure  our  progress  and  effec¬ 
tiveness  in  getting  there.  If  we  don’t  take 
the  time  to  determine  precisely  where 
we  are  going,  we  will  not  be  able  to  de¬ 
termine  the  effectiveness  of  our  process. 
And  as  the  saying  goes,  “  Any  road  will 
do  if  you  don’t  know  where  you  are  go¬ 
ing.” 

The  ISD  model  is  especially  effective 
in  developing  learning  experiences  that 
meet  the  needs  of  a  well-defined  target 
audience  and  other  stakeholders  (such  as 
our  Functional  Boards).  It  can  do  so  while 
maintaining  currency  and  consistency  in 
a  rapidly  changing  environment,  such  as 
acquisition  reform. 

For  the  past  year,  the  Defense  Systems 
Management  College  (DSMC),  as  well  as 
the  other  DAU  Consortium  Schools,  has 
performed  in  a  yeoman  fashion  the 
reengineering  of  all  of  its  major  courses. 

The  process  the  schools  employed  was 
the  DAU  ISD  model.  During  this  process, 
the  schools  experienced  a  considerable 
amount  of  oversight  from  the  Functional 
Boards  and  DAU.  For  many  in  the  con¬ 
sortium  schools,  this  has  been  a  trau¬ 
matic  experience.  ISD  in  the  best  of 
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opment  for  faculty  and  staff.  He  holds  a  B.S.  degree  in  biology,  an  M.Ed.  degree  in  counselor 
education  from  the  University  of  Virginia,  and  an  Ed.D.  degree  in  higher  education  administra¬ 
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circumstances  is  a  labor-intensive  process 
requiring  between  60  to  100  hours  of  de¬ 
velopment  time  for  every  hour  in  the  class¬ 
room.  This  development  overhead  rapidly 
increases  as  we  move  to  the  technology- 
based  education  (TBE)  delivery  media. 

For  the  novice,  the  gains  in  quality 
learning  by  using  this  process  are  not 
immediately  apparent.  It  is  especially  dif¬ 
ficult  to  see  the  worth  of  this  effort  when 
the  immediate  results  of  the  first  offering 
are  not  showing  the  gains  in  student  satis¬ 
faction  that  was  and  should  be  expected. 
Never  fear;  ISD  is  an  iterative  process  and 
is  expected  to  be  less  than  perfect  on  the 
first  offering.  The  truth  be  known,  it  is 
never  finished.  The  final  step  in  the  ISD 
process  is  to  evaluate  the  process  (note  that 
I  did  not  say  “the  student”)  and  change 
the  process  when  indicated.  If  you  make 
a  conscious  decision  to  forego  the  “try 
out  with  revision”  step  (also  known  as  a 
pilot),  then  you  can  expect  even  greater 
discrepancies. 

ISD,  as  an  engineering  process,  appears 
to  have  little  flexibility.  Some  may  view 
the  process  as  a  loss  of  academic  freedom. 
Academic  freedom,  as  defined  by 
Brubacher  &  Rudy  (1976,  p.  308)  is... 
“the  right  of  a  professor  to  follow  an 
argument  whither  so  ever  it  may  lead 
either  in  his  research  or  in  his  teaching...  ” 
Another  definition,  espoused  by  Webster's 
//( 1984,  p.  69),  is  “. .  .without  interference, 
as  from  school  or  public  officials.”  These 
definitions  are  the  quintessential  essence 
of  higher  education  , where  ideas  are 
allowed  to  be  discussed  and  aired  with¬ 
out  fear  of  retribution  by  higher  author¬ 
ity.  Academic  freedom  at  its  core  as¬ 
sures  First  Amendment  rights  and  fosters 
research,  creativity,  and  learning  by  allow¬ 
ing  the  full  range  of  the  exploration  of 


ideas.  Neither  Webster's  nor  Brubacher  & 
Rudy’s  definition  of  academic  freedom, 
however,  exempts  a  professor  from  meet¬ 
ing  the  specific  and  implicit  objectives  of 
the  course  or  lesson,  as  articulated  in  the 
performance  outcomes  and  the  terminal 
learning  objectives  (TLOs).  It  is  an 
institution’s  prerogative  to  determine 
objectives.  One  implicitly  agrees  to  follow 
this  direction  when  hired. 

One  of  the  primary  strengths  of  the  sys¬ 
tems  approach  to  developing  curriculum 
is  defining  clear 
and  measurable 

objectives.  By  "One  of  the 

clearly  defining  primary  strengths 

the  objectives  of  the  systems  a 

and  the  assess-  pproach  to  develop¬ 
ment  processes  *"9  curriculum  I* 

of  a  course  or  dining  dear 

vides  a  consis¬ 
tent  and  repeat- 

able  educational  experience.  Consistency 
is  the  sense  that  any  number  of  students 
can  be  exposed  to  the  process  and  be  as¬ 
sured  that  they  will  attain  mastery  of  the 
subject.  Constructed  properly,  this  consis¬ 
tency  is  assured  by  designing  fidelity  into 
the  lesson  or  course  from  a  test  question 
up  to  and  including  the  performance  out¬ 
come.  In  simpler  terms,  it  means  that  each 
question  on  a  test,  and  each  assessment 
opportunity,  is  designed  to  address  the  be¬ 
haviors  expressed  in  the  performance  out¬ 
come  with  a  high  level  of  correlation.  This 
characteristic  is  extremely  important  to  an 
institution  that  needs  to  educate  large  num¬ 
bers  of  students  to  meet  an  acceptable  per- 
formance  standard.  The  consortium 
schools  are  such  institutions. 

Another  concern  of  the  faculty  is  the 
expenditure  of  time  needed  to  design 
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courses  using  the  systematic  approach 
models.  Earlier  I  stated  that,  for  planning 
purposes  ,  an  expenditure  of  60  to  100 
hours  of  development  time  is  required 
for  each  hour  in  the  classroom.  In  my 
experience  these  hours  increase  dramati¬ 
cally  when  we  are  developing  or  convert¬ 
ing  courses  to  TBE.  Time  is  a  scarce  and 
critical  commodity,  especially  as  the 
demand  for  professor  time  is  on  the  rise 
while  personnel  resources  are  being 
reduced. 


What  I  am  concerned  about  is  the  pos¬ 
sibility  that  the  pressures  related  to  teach¬ 
ing  preparation,  teaching,  and  curriculum 
development  are  so  great  that  the  true 
value  of  what  the  schools  have  accom¬ 
plished  will  not 


"What  l  am 
concerned  about 
is  the  possibility 
that  the  pressures 
related  to  teaching 
preparation,  teach¬ 
ing,  and  curriculum 
development  are  so 
great  that  the  true 
value  off  what  the 
schools  have  accom¬ 
plished  will  not  be 
understood  by  the 
faculty," 


be  understood 
by  the  faculty. 
In  this  environ¬ 
ment  there  is  a 
possibility  that 
the  ISD  process 
will  be  regarded 
as  noxious  and 
with  little  return 
on  investment. 


noois  nave  accom-  The  result  may 

lished  will  net  be  be  the  paying 

nderstoed  by  the  0f  lip  service  to 

iculty."  fU 

9  the  process  or 

worse  yet,  re¬ 
turning  to  the  former  “intuitive”  curricu¬ 
lum  design  method — so  long  practiced 
here  and  in  higher  education  in  general. 
(By  “intuitive”  I  mean  each  individual  pro¬ 
fessor  selecting  what  is  important  to  learn, 
resulting  in  the  lack  of  consistency  among 
and  between  professors  of  the  same  sub¬ 
ject  and  over  time.)  If  this  occurred,  it 
would  be  a  disservice  to  the  schools  and 
to  all  their  customers. 


Intuitive  curriculum  development  has 
found  a  legitimate  place  in  traditional 
higher  education.  Higher  education,  (ex¬ 
cept  for  some  technical  curricula,  such  as 
nursing),  is  not  responsible  for  educating 
students  for  a  specific  workplace  with  spe¬ 
cific  expected  behaviors.  Therefore,  con¬ 
sidering  the  vast  amount  of  knowledge 
accumulated  in  any  traditional  field,  such 
as  liberal  arts  or  science,  the  selection  of 
the  outcomes  is  usually  prescribed  by  the 
individual  professor.  Those  professors 
cannot  measure  their  educational  effec¬ 
tiveness  against  a  set  of  competencies  or 
performance  measures,  because  those 
competencies  and  performance  measures 
do  not  exist.  Who,  for  example,  knows 
where  student  “X”  is  going  to  work  when 
he  or  she  graduates  with  a  B.S.  degree 
from  Anyplace  University?  The  graduates 
themselves  do  not  know  until  the  final 
hour,  if  then.  Traditional  higher  education 
measures  student  against  student.  The  stu¬ 
dents  compete  for  a  grade  and  are  not  en¬ 
couraged  to  engage  in  cooperative  learn¬ 
ing.  In  acquisition  management  team 
problem  solving,  integrated  product  teams 
(for  example)  are  not  only  allowed,  they 
are  mandated.  The  schools  teaching  ac¬ 
quisition  management  must  simulate  the 
work  environment;  therefore  learning,  like 
work,  is  cooperative.  Students  should  not 
be  measured  against  each  other  but 
assessed  against  a  performance  standard. 

We  in  the  Consortium  Schools  know  the 
specific  workplace  where  our  graduates 
are  going  and  how  they  will  be  required 
to  perform.  Therefore,  our  course  objec¬ 
tives — performance  standards  if  you 
will — cease  to  be  the  prerogative  of  the 
individual  professor  but  become  the  natu¬ 
ral  outcome  of  a  systematic  analysis  of  the 
student  and  institutional  needs. 
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Figure  1.  The  Systems  Approach  to  Developing  Curriculum 
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Please  understand  that  the  ISD  process 
only  requires  that  the  student  reaches  mini¬ 
mum  behaviors  to  perform  adequately 
in  the  field  through  the  educational  pro¬ 
cess.  This  does  not  prohibit  the  profes¬ 
sor  from  facilitating  learning  to  the  ex¬ 
tent  a  student’s  prior  experiences  and  time 
allows.  Additionally,  the  ISD  process  will 
provide  one  way  of  teaching  a  course  via 
the  Instructor  Guide  (IG).  The  IG  should 
never  be  the  only  way  to  teach.  Therefore 
an  individual  professor  maintains  the  right 
to  select  the  media  and  to  go  beyond  the 
minimum  objectives. 

The  school’s  responsibility  is  to  take  the 
institutional  needs  as  defined  by  the  Func¬ 
tional  Boards  (Performance  Outcomes), 
and  design  a  course  that  contains  termi¬ 
nal  learning  objectives  (TLOs)  and  a  se¬ 
lection  of  enabling  learning  objectives 
(ELOs),  a  student  and  course  assessment 

plan,  and  a  de- 


"The  delivery  and 
assessment  areas 
are  most  appropri¬ 
ate  for  individual 
professors  to  exer¬ 
cise  academic  free¬ 
dom  and  creativity 
In  making  the  learn* 


adult  and  creative 
experience  for  the 
student/' 


livery  system, 
which  meets  the 
level  of  learning 
desired  within 
the  constraints 
of  time  and  re¬ 
sources.  The  de¬ 
livery  and  as¬ 
sessment  areas 
are  most  appro¬ 
priate  for  indi¬ 
vidual  profes¬ 


sors  to  exercise 
academic  freedom  and  creativity  in  mak¬ 
ing  the  learning  experience  an  adult  and 
creative  experience  for  the  student.  Each 
offering  of  a  class  need  not  be  a  carbon 


copy  of  the  others.  The  professors’  indi¬ 
vidual  creative  strengths  and  specific 
needs  of  their  students  drive  the  method¬ 
ology.  The  only  caveat  expressed  is  that 
when  a  professor  deviates  from  a  course 
design,  his  or  her  students  must  fare  as 
well  as  others,  as  measured  by  standard 
assessment  tools. 

I  believe  that  if  we  continue  to  follow  a 
systems  approach  design  philosophy  and 
exploit  the  inexhaustible  talent  of  the  Con¬ 
sortium  Schools  faculty,  we  can  meet  the 
spirit  and  the  letter  of  the  D  AU  guidance 
and  encourage  creativity  while  bringing  a 
dynamic  and  challenging  learning  climate 
to  our  students  Figure  1  is  a  notional  flow 
of  a  typical  systems  approach  to  designing 
curricula. 

Let  us  remember  that  what  we  have 
done  in  this  last  year,  as  far  as  curriculum 
design  is  concerned,  is  to  utilize  with  some 
degree  of  efficiency  what  has  long  been 
proven  by  educators  and  trainers  alike  to 
be  an  effective  methodology  to  design  and 
improve  curriculum.  It  is  not  the  only 
way— but  it  is  a  proven  way.  Let  us  not 
be  discouraged  by  a  lack  of  empirical  data 
to  prove  our  success.  That  will  come  from 
the  assessment  data  gathered  from  stu¬ 
dents,  faculty,  alumni,  and  their  supervi¬ 
sors.  Let  us  not  fail  to  realize  its  benefits 
to  us  and,  more  important,  to  those  whom 
we  are  most  responsible:  the  students. 
Teachers  are  ultimately  responsible  for  fa¬ 
cilitating  the  learning  environment.  We 
have  an  approved  and  a  provable  design 
methodology.  Let’s  not  throw  the  baby  out 
with  the  bath  water! 
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process.  Defense  acquisition  is  defined  as 
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other  systems,  supplies,  or  services  to  sat¬ 
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tended  for  use  in  support  of  military  mis¬ 
sions. 


Research  Articles 


Manuscripts  should  reflect  research  or 
empirically-supported  experience  in  one 
or  more  of  the  aforementioned  areas  of 
acquisition.  Research  or  tutorial  articles 
should  not  exceed  4,500  words.  Opinion 
pieces  should  be  limited  to  1,500  words. 

We  publish  Defense  Acquisition  re¬ 
search  articles  that  involve  systemic  in¬ 


quiry  into  a  significant  research  question. 
The  article  must  produce  a  new  or  revised 
theory  of  interest  to  the  acquisition  com¬ 
munity.  You  must  use  a  reliable,  valid  in¬ 
strument  to  provide  your  measured  out¬ 
comes. 


Manuscript  Sections 


The  introduction  should  state  the  pur¬ 
pose  of  the  article  and  concisely  summa¬ 
rize  the  rationale  for  the  undertaking. 

The  methods  section  should  include  a 
detailed  methodology  that  clearly  de¬ 
scribes  work  performed.  Although  it  is 
appropriate  to  refer  to  previous  publica¬ 
tions  in  this  section,  the  author  should  pro¬ 
vide  enough  information  so  that  the  expe¬ 
rienced  reader  need  not  read  earlier  works 
to  gain  understanding  of  the  methodology. 

The  results  section  should  concisely 
summarize  findings  of  the  research  and 
follow  the  train  of  thought  established  in 
the  methods  section.  This  section  should 
not  refer  to  previous  publications,  but 
should  be  devoted  solely  to  the  current 
findings  of  the  author. 

The  discussion  section  should  empha¬ 
size  the  major  findings  of  the  study  and 
its  significance.  Information  presented  in 
the  aforementioned  sections  should  not  be 
repeated. 
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Research  Considerations _ _ 

Contributors  should  also  consider  the 
following  questions  in  reviewing  their  re- 
search-based  articles  prior  to  submission: 

•  Is  the  research  question  significant? 

•  Are  research  instruments  reliable  and 
valid? 

•  Are  outcomes  measured  in  a  way 
clearly  related  to  the  variables  under 
study? 

•  Does  the  research  design  fully  and  un¬ 
ambiguously  test  the  hypothesis? 

•  Did  you  build  needed  controls  into  the 
study? 

Contributors  of  research-based  submis¬ 
sions  are  also  reminded  they  should  share 
any  materials  and  methodology  necessary 
to  verify  their  conclusions. 

Critiwa  for  Tutorials _ 

Tutorials  should  provide  special  in¬ 
struction  or  knowledge  relevant  to  an  area 
of  defense  acquisition  to  inform  the  De¬ 
fense  Acquisition  Workforce. 

Topics  for  submissions  should  rely  on 
or  be  derived  from  observation  or  experi¬ 
ment,  rather  than  theory.  The  submission 
should  provide  knowledge  in  a  particular 
area  for  a  particular  purpose. 


Opiwoh  Criteria _ 

Opinion  articles  should  reflect  judg¬ 
ments  based  on  the  special  knowledge  of 
the  expert.  Opinion  articles  should  be 
based  on  observable  phenomena  and  pre¬ 
sented  in  a  factual  manner;  that  is,  sub¬ 
missions  should  imply  detachment.  The 
observation  and  judgment  should  not  re¬ 
flect  the  author’s  personal  feelings  or 
thoughts.  Nevertheless,  opinion  pieces 
should  clearly  express  a  fresh  point  of 
view,  rather  than  negatively  criticize  the 
view  of  another  previous  author. 

Manuscript  Style _ _ 

We  will  require  you  to  recast  your  last 
version  of  the  manuscript,  especially  ci¬ 
tations  (e.g.,  footnotes  or  endnotes)  into 
the  format  required  in  two  specific  style 
manuals.  The  ARQ  follows  the  author 
(date)  form  of  citation.  We  expect  you  to 
use  the  Publication  Manual  of  the  Ameri¬ 
can  Psychological  Association  (4th  Edi¬ 
tion),  and  the  Chicago  Manual  of  Style 
(14th  Edition).  The  ARQ  follows  the  au¬ 
thor  (date)  form  of  citation. 

Contributors  are  encouraged  to  seek  the 
advice  of  a  reference  librarian  in  complet¬ 
ing  citations  of  government  documents. 
Standard  formulas  of  citations  may  give 
only  incomplete  information  in  reference 
to  government  works.  Helpful  guidance 
is  also  available  in  Gamer,  D.L.  and  Smith, 
D.H.,  1993,  The  Complete  Guide  to  Cit¬ 
ing  Government  Documents:  A  Manual 
for  Writers  and  Librarians  (Rev.  Ed.), 
Bethesda,  MD:  Congressional  Informa¬ 
tion  Service,  Inc. 
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Copyright  Information 


The  ARQ  is  a  publication  of  the  United 
States  Government  and  as  such  is  not 
copyrighted.  Contributors  of  copyrighted 
works  and  copyright  holders  of  works  for 
hire  are  strongly  encouraged  to  request 
that  a  copyright  notification  be  placed  on 
their  published  work  as  a  safeguard  against 
unintentional  infringement.  The  work  of 
federal  employees  undertaken  as  part  of 
their  official  duties  is  not  subject  to  copy¬ 
right. 

In  citing  the  work  of  others,  it  is  the 
contributor’s  responsibility  to  obtain  per¬ 
mission  from  a  copyright  holder  if  the  pro¬ 
posed  use  exceeds  the  fair  use  provisions 
of  the  law  (see  U.S.  Government  Printing 
Office,  1994,  Circular  92:  Copyright  Law 
of  the  United  States  of  America,  p.  15, 
Washington,  DC:  Author).  Contributors 
will  be  required  to  submit  a  copy  of  the 
written  permission  to  the  editor  before 
publication. 


Manuscript  Format 


Pages  should  be  double-spaced  and  or¬ 
ganized  in  the  following  order:  title  page, 
abstract,  body,  reference  list,  author’s  note 
(if  any),  and  figures  or  tables.  To  ensure 
anonymity,  each  paper  should  be  submit¬ 
ted  with  a  separate  page  that  includes  the 
author(s)’s  name(s)  and  complete  address, 
and  the  paper  should  include  the  title,  ab¬ 
stract,  keywords,  body,  complete  set  of 
references,  along  with  tables  and  figures 
at  the  end.  Authors  are  reminded  not  to 
refer  to  themselves  or  to  their  own  work 
directly  in  the  paper.  Figures  or  tables 
should  not  be  inserted  (or  embedded,  etc.) 


into  the  text,  but  segregated  one  to  a  page 
following  the  text.  Articles  must  be  print¬ 
able  within  one  issue  and  should  not  ex¬ 
ceed  4,500  words  for  research  or  tutorials 
and  1,500  words  for  opinion  pieces;  ar¬ 
ticles  will  not  be  printed  in  parts  or  in  a 
continuing  series.  If  material  is  submitted 
on  a  computer  diskette,  each  figure  or  table 
should  be  recorded  in  a  separate,  export¬ 
able  file  (i.e.,  a  readable  .eps  file).  For 
additional  information  on  the  preparation 
of  figures  or  tables,  see  CBE  Scientific 
Illustration  Committee,  1988,  Illustrating 
Science:  Standards  for  Publication, 
Bethesda,  MD:  Council  of  Biology  Edi¬ 
tors,  Inc.  Please  restructure  briefing  charts 
and  slides  to  a  look  similar  to  those  in  pre¬ 
vious  issues  of  ARQ. 

The  author  (or  corresponding  author  in 
the  case  of  multiple  authorship)  should 
attach  to  the  manuscript  a  signed  cover 
letter  that  provides  the  author’s  name,  ad¬ 
dress,  and  telephone  number  (fax  and 
Internet  addresses  are  also  appreciated). 
The  letter  should  verify  that  the  submis¬ 
sion  is  an  original  product  of  the  author; 
that  it  has  not  been  published  before;  and 
that  it  is  not  under  consideration  by  an¬ 
other  publication.  Details  about  the  manu¬ 
script  should  also  be  included  in  this  let¬ 
ter:  for  example,  its  title,  word  length,  the 
need  for  copyright  notification,  the  iden¬ 
tification  of  copyrighted  material  for 
which  permission  must  be  obtained,  a  de¬ 
scription  of  the  computer  application  pro¬ 
grams  and  file  names  used  on  enclosed 
diskettes,  etc. 

The  letter,  one  copy  of  the  printed 
manuscript,  and  any  diskettes  should  be 
sturdily  packaged  and  mailed  to:  Defense 
Systems  Management  College,  Attn: 
DSMC  Press  (ARQ),  9820  Belvoir  Road, 
Suite  3,  Fort  Belvoir,  VA  22060-5565. 


397 


Acquisition  Review  Quarterly— Fall  1998 


In  most  cases,  the  author  will  be  noti¬ 
fied  that  the  submission  has  been  received 
within  48  hours  of  its  arrival.  Following 
an  initial  review,  submissions  will  be  re¬ 
ferred  to  referees  and  subsequent  consid¬ 
eration  by  the  ARQ  Editorial  Board. 

Contributors  may  direct  their  questions 
to  the  Editor,  ARQ ,  at  the  address  shown 


above,  by  calling  (703)  805-4290  (fax 
805-  2917),  or  via  the  Internet  at: 
gonzalezd@dsmc.dsm.mil. 

The  DSMC  Home  Page  can  be  accessed 
at: 

http://www.dsmc.dsm.mil. 
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